FisherDistill: Diffusion Distribution Distillation via Closed-Form Fisher Critics
AuthorsMahesh Bhosale, Yuanhao Zhai, Siwei Lyu, Junsong Yuan, David Doermann, Defang Chen
AffiliationsUniversity at Buffalo, SUNY · Meta · University of California, Berkeley
FisherDistill uses a closed-form statistical critic to improve fast diffusion generators without training an adversarial discriminator.
Key results
Down from 1.42; recall rises from 0.60 to 0.62.
Down from 8.58 after Fisher refinement.
Down from 28.94 for the SiD-DiT baseline.
Improves on the Stage-II baseline score of 30.20.
FisherDistill reaches its best FID versus 4.93M samples for matched GAN refinement.
What the paper found
FisherDistill offers a non-adversarial way to close the quality gap in one-step diffusion models. Instead of training a discriminator, it uses real and generated samples’ intermediate features from a frozen teacher to calculate a closed-form Fisher direction, accounting for both the difference in feature means and their shared covariance. This critic adds a corrective signal to existing distillation losses, without learnable critic parameters or a separate adversarial optimization loop. Across DMD2, SiD, SiD-DiT, and 1.x-Distill, the method improves generation quality: on ImageNet-64, one-step DMD2 lowers FID from 1.42 to 1.40 while raising recall from 0.60 to 0.62; on SD v1.5, FID falls from 8.58 to 8.46; and on SANA-0.6B, it drops from 28.94 to 25.24. For 1.x-Distill, FID improves from 30.20 to 29.54. FisherDistill also reaches its best DMD2 FID after 1.34M generated samples, compared with 4.93M for the matched GAN refinement. The paper states that Meta was not involved in the work.
Original abstract
Recent advances in diffusion distillation have enabled high-quality image generation in only one or a few network function evaluations (NFEs). Despite their efficiency, aggressively distilled generators often retain a residual distribution gap from their multi-step teachers, motivating recent methods to introduce adversarial refinement with a learned discriminator. In this work, we ask whether such refinement necessarily requires a learned adversarial discriminator. We introduce \textbf{FisherDistill}, a simple alternative based on Fisher linear discriminant analysis. Using intermediate representations from the frozen teacher, we estimate the first- and second-order statistics of real and generated samples and derive a closed-form Fisher direction that maximally separates their feature distributions relative to within-class variation. This direction provides a lightweight refinement signal that complements the original distillation objective without requiring an additional adversarial discriminator or its optimization loop. Across multiple distillation frameworks, our approach consistently improves generation quality over their respective distilled baselines and substantially closes the gap to adversarially refined counterparts. Code: https://bhosalems.github.io/FisherDistill/
Read the original paperMore in Diffusion Models
Browse all 82 papers →FewStepRL: Fine-tuning Few-step Diffusion Models with Reinforcement Learning
Haitao Lin, Minsi Ren, Yang Li, Xingtong Ge, Zheyuan Hu, Peiyan Hu, Tailin Wu
FewStepRL adapts multiple reinforcement-learning fine-tuning methods to make reward-driven improvements to fast, few-step diffusion models.
JVT: Just Video Transformer makes pixel-space video diffusion simple
Jianjun Zhou, Hua Geng, Haoyu Guo
JVT shows that a straightforward Transformer can generate video directly in pixels—and even produce matching depth-like geometry alongside it.
Restoring Diversity in Distilled Image and Video Generators by Probing Activations
Furkan Horoz, Ozgur Kara, Kaan Oktay, Adil Kaan Akan, Atakan Kara, Anand Bhattad, James Matthew Rehg
The paper restores variety in fast distilled image and video generators by finding and correcting a collapse in their internal activation directions.