Seventy3: 用NotebookLM将论文生成播客，让大家跟着AI一起进步。 今天的主题是：Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You ThinkMain Theme: This paper introduces REPresentation Alignment (REPA), a novel technique for accelerating and improving the training of diffusion transformers for image generation by aligning their internal representations with high-quality, pre-trained visual representations from self-supervised learning models. Key Findings: Diffusion models learn discriminative representations, but they lag behind dedicated self-supervised methods: While analyzing SiT and DiT models, the authors observed that their hidden states contain semantically meaningful information (demonstrated by linear probing). However, their performance on image classification tasks falls significantly short of models like DINOv2. Weak alignment exists between diffusion model representations and self-supervised representations: Using CKNNA, a representation alignment metric, the authors revealed a weak alignment between diffusion models and DINOv2, suggesting room for improvement. REPA effectively bridges this representation gap: By regularizing the diffusion model to align its hidden states with pre-trained representations (e.g., DINOv2) of clean images, REPA significantly boosts training efficiency and final generation quality. This is evident in improved FID scores, faster convergence, and better linear probing accuracy.Most Important Ideas and Facts: REPA Mechanism: REPA works by maximizing the similarity between a projection of the noisy input's hidden state in the diffusion model and the pre-trained representation of the corresponding clean image. This encourages the diffusion model to learn noise-invariant, semantically rich features early on. "REPA distills the pretrained self-supervised visual representation y∗ of a clean image x into the diffusion transformer representation h of a noisy input x̃." Impact on Training Efficiency: REPA significantly accelerates training convergence. Notably, SiT-XL/2 with REPA achieved an FID of 7.9 in just 400K iterations, surpassing the vanilla SiT-XL/2 trained for 7M iterations. This translates to a >17.5x speedup. "Notably, model training becomes significantly more efficient and effective, and achieves >17.5× faster convergence than the vanilla model." Improved Generation Quality: REPA consistently improves FID scores across different model sizes and architectures. For SiT-XL/2, REPA achieved a state-of-the-art FID of 1.42 with guidance interval scheduling, outperforming existing diffusion models. "In terms of final generation quality, our approach achieves state-of-the-art results of FID=1.42 using classifier-free guidance with the guidance interval." Targeted Regularization: Applying REPA to only the first few transformer blocks proves most effective, allowing later layers to focus on refining high-frequency details based on the already aligned representations. "Interestingly, with REPA, we observe that sufficient representation alignment can be achieved by aligning only the first few transformer blocks." Stronger Encoders Yield Better Results: Utilizing more powerful pre-trained encoders as the target representation consistently leads to improved generation and linear probing results, highlighting the importance of high-quality representations. "When a diffusion transformer is aligned with a pretrained encoder that offers more semantically meaningful representations (i.e., better linear probing results), the model not only captures better semantics but also exhibits enhanced generation performance"Quotes: Figure 1 caption: "Representation alignment makes diffusion transformer training significantly easier. Our framework, REPA, explicitly aligns the diffusion model representation with powerful pretrained visual representation through a simple regularization. Notably, model training becomes significantl