Nemotron-Labs-Diffusion-Uni: Elastic Self-Refining Unified Diffusion LLM with Cross-Paradigm On-Policy Distillation
AuthorsShufan Li, Hanrong Ye, Greg Heinrich, Yonggan Fu, Haotian Zhang, Sifei Liu, Aditya Grover, Pavlo Molchanov
Affiliations*Core Contribution
NLD-Uni uses self-refining diffusion and cross-paradigm distillation to bring visual understanding and image generation together in one language model.
Key results
Average across the reported image-understanding benchmarks.
Text-to-image generation score.
Human-preference image-quality score; higher than Qwen-Image’s 10.91.
Image-editing score, compared with GPT-4o’s 4.20.
Multimodal reasoning score after cross-paradigm distillation.
Multimodal mathematical-reasoning score after cross-paradigm distillation.
What the paper found
NVIDIA’s NLD-Uni is a unified discrete diffusion model for visual understanding and image generation, built around an elastic mixture-of-transformers architecture that loads task-specific parameter branches. Its first key innovation, cross-paradigm on-policy distillation, transfers reasoning from the autoregressive Nemotron-3 Nano Omni and image-editing skill from continuous-flow Qwen-Image; for understanding, it aligns the model’s diffusion predictions with its own autoregressive predictions to avoid conflicting teacher signals. Its second innovation is progressive self-refining diffusion: instead of replacing image tokens directly with masks, it corrupts them through intermediate states organized by a ball tree over VQ-code embeddings, enabling iterative token correction during generation. NLD-Uni reaches 78.3 average accuracy across visual-understanding benchmarks, scores 0.89 on GenEval and 11.12 on HPSv3, and earns 4.17 on ImgEdit, close to GPT-4o’s 4.20. Cross-paradigm distillation raises its MathVista score to 77.0 and We-Math to 74.9, while HPSv3 exceeds Qwen-Image’s 10.91. Together, the results show how one diffusion model can combine multimodal reasoning, grounding, image generation, and editing while narrowing the quality gap with specialized models.
Original abstract
We propose Nemotron-Labs-Diffusion-Uni (NLD-Uni), a unified discrete diffusion large language model (DLLM) for visual understanding and generation tasks. Compared with prior unified diffusion models, NLD-Uni introduces two major innovations. First, we propose a unified cross-paradigm on-policy distillation (OPD) framework that successfully transfers the visual understanding ability of autoregressive VLMs and visual generation ability of continuous flow models, to a unified DLLM. Second, NLD-Uni incorporates a novel progressive self-refining diffusion process that gradually corrupts each token across multiple steps during training, as opposed to adopting the conventional masked diffusion formulation that directly corrupts a clean image token to an uninformative mask token. At inference, this design allows NLD-Uni to make intermediate commit to a token, enabling progressive self-refinement for better visual quality. Through extensive experiments, we show that NLD-Uni achieves state-of-the-art performance among unified DLLMs on diverse tasks including visual question-answering, multi-modal reasoning, object grounding, text-to-image generation, and image editing.
Read the original paperMore in Diffusion Models
Browse all 73 papers →On the SPOT: Enhancing Speculative Decoding for Diffusion Language Models via Commit-Order Prediction
Eliezer Shlomi, Eliron Rahimi, Avi Mendelson, Chaim Baskin
SPOT speeds up diffusion language models by predicting which positions to commit next, then checking multiple proposed tokens in one pass.
PhysLDM: Latent Diffusion for High-Fidelity Deformable Simulation
Yu Zhang, Xudong Xu, Xingang Pan
PhysLDM uses latent diffusion to predict plausible long-term motion of complex deformable objects more effectively than deterministic averaging.
SWIFT: Self-guided Weak-to-strong dIFfusion models via Target extrapolation
Yanjin He, Cuifeng Shen, Wanzhou Lei, Maohua Li, Tao Lan, Kan Liu, Hanlin Tang, Qing Qu
SWIFT uses a weaker diffusion model to help train a stronger one, aiming for better image generation without extra inference-time cost.