Restoring Diversity in Distilled Image and Video Generators by Probing Activations
AuthorsFurkan Horoz, Ozgur Kara, Kaan Oktay, Adil Kaan Akan, Atakan Kara, Anand Bhattad, James Matthew Rehg
AffiliationsUniversity of Illinois at Urbana-Champaign · fal
The paper restores variety in fast distilled image and video generators by finding and correcting a collapse in their internal activation directions.
Key results
ReDiverse score; the four-step student scored 59.4.
ReDiverse score; the four-step student scored 48.4.
ReDiverse score; the four-step student scored 31.6.
ReDiverse score; the four-step student scored 36.5.
What the paper found
Four-step distilled generators can produce similar outputs from different random seeds, but ReDiverse finds the cause is not a lack of activation variance: at one intermediate block during the first denoising step, students concentrate that variance into fewer directions than their teachers. The method repairs this bottleneck in two ways: ReDiverse-FT trains localized LoRA adapters to match teacher activations, while training-free ReDiverse-TF uses PCA to inject variation along teacher directions. Tested on SDXL, SD3, Wan 2.1, and MiniMax-H3 using GenEval and VBench prompts, the combined method raises DINO diversity from 59.4 to 65.0 on SDXL, 48.4 to 60.1 on SD3, 31.6 to 52.2 on Wan 2.1, and 36.5 to 50.9 on MiniMax-H3, generally retaining generation quality and prompt alignment. A user study and Gemini 3.1 Pro evaluations also favored ReDiverse over the few-step student methods for diversity. The central insight is that downstream network blocks can still render structural variation once the missing teacher directions are restored.
Original abstract
Distilling diffusion models into few-step student models substantially reduces sampling latency, but induces severe diversity collapse: samples generated from different random seeds show minimal variation. In this work, we investigate the root cause of this failure by probing the internal representations of student models relative to their teachers. We find that distilled students maintain total activation variance comparable to their teachers, but concentrate it into substantially fewer directions. This directional collapse is localized to an identifiable intermediate block at the initial denoising step. Consequently, diversity collapse stems from a misallocation of activation variance rather than an inherent loss of generative capacity. Guided by this finding, we propose ReDiverse to reinstate the missing variance directions. At training time, ReDiverse-FT fine-tunes a lightweight adapter that regresses student activations onto the teacher's, reviving the teacher's feature variance characteristic. At inference time, ReDiverse-TF introduces a training-free mechanism that steers activations along precomputed teacher principal directions. Across image and video distillation benchmarks, ReDiverse consistently restores sample diversity while preserving generation quality, further supported by the user study.
Read the original paperMore in Diffusion Models
Browse all 82 papers →FewStepRL: Fine-tuning Few-step Diffusion Models with Reinforcement Learning
Haitao Lin, Minsi Ren, Yang Li, Xingtong Ge, Zheyuan Hu, Peiyan Hu, Tailin Wu
FewStepRL adapts multiple reinforcement-learning fine-tuning methods to make reward-driven improvements to fast, few-step diffusion models.
JVT: Just Video Transformer makes pixel-space video diffusion simple
Jianjun Zhou, Hua Geng, Haoyu Guo
JVT shows that a straightforward Transformer can generate video directly in pixels—and even produce matching depth-like geometry alongside it.
BYOD: Build Your Own DLM - Efficient Autoregressive-to-Diffusion Language Model Conversion
Ruurd J. A. Kuiper, Robert A. Bagheri
BYOD shows how to turn existing language models into diffusion generators with small LoRA adapters, while highlighting the quality gap that remains.