NTH

FisherDistill: Diffusion Distribution Distillation via Closed-Form Fisher Critics

AuthorsMahesh Bhosale, Yuanhao Zhai, Siwei Lyu, Junsong Yuan, David Doermann, Defang Chen

AffiliationsUniversity at Buffalo, SUNY · Meta · University of California, Berkeley

November 10, 2026 2 min read
Watch on YouTube
The one-line take

FisherDistill uses a closed-form statistical critic to improve fast diffusion generators without training an adversarial discriminator.

Key results

1.40
ImageNet-64 DMD2 FID

Down from 1.42; recall rises from 0.60 to 0.62.

8.46
SD v1.5 DMD2 FID

Down from 8.58 after Fisher refinement.

25.24
SANA-0.6B FID

Down from 28.94 for the SiD-DiT baseline.

29.54
1.x-Distill FID

Improves on the Stage-II baseline score of 30.20.

1.34M
Samples to best DMD2 FID

FisherDistill reaches its best FID versus 4.93M samples for matched GAN refinement.

What the paper found

FisherDistill offers a non-adversarial way to close the quality gap in one-step diffusion models. Instead of training a discriminator, it uses real and generated samples’ intermediate features from a frozen teacher to calculate a closed-form Fisher direction, accounting for both the difference in feature means and their shared covariance. This critic adds a corrective signal to existing distillation losses, without learnable critic parameters or a separate adversarial optimization loop. Across DMD2, SiD, SiD-DiT, and 1.x-Distill, the method improves generation quality: on ImageNet-64, one-step DMD2 lowers FID from 1.42 to 1.40 while raising recall from 0.60 to 0.62; on SD v1.5, FID falls from 8.58 to 8.46; and on SANA-0.6B, it drops from 28.94 to 25.24. For 1.x-Distill, FID improves from 30.20 to 29.54. FisherDistill also reaches its best DMD2 FID after 1.34M generated samples, compared with 4.93M for the matched GAN refinement. The paper states that Meta was not involved in the work.

Original abstract

Recent advances in diffusion distillation have enabled high-quality image generation in only one or a few network function evaluations (NFEs). Despite their efficiency, aggressively distilled generators often retain a residual distribution gap from their multi-step teachers, motivating recent methods to introduce adversarial refinement with a learned discriminator. In this work, we ask whether such refinement necessarily requires a learned adversarial discriminator. We introduce \textbf{FisherDistill}, a simple alternative based on Fisher linear discriminant analysis. Using intermediate representations from the frozen teacher, we estimate the first- and second-order statistics of real and generated samples and derive a closed-form Fisher direction that maximally separates their feature distributions relative to within-class variation. This direction provides a lightweight refinement signal that complements the original distillation objective without requiring an additional adversarial discriminator or its optimization loop. Across multiple distillation frameworks, our approach consistently improves generation quality over their respective distilled baselines and substantially closes the gap to adversarially refined counterparts. Code: https://bhosalems.github.io/FisherDistill/

Read the original paper

More in Diffusion Models

Browse all 82 papers →