Large Language Continuous Diffusion Models
AuthorsZhihan Yang, Wei Guo, Jean-Marie Lemercier, Simon Welker, Yonggan Fu, Mohammad Mahdi Kamani, Sajad Norouzi, Julius Berner, Tomas Geffner, Karsten Kreis, Yongxin Chen, Molei Tao, John Thickstun, Pavlo Molchanov, Ante Jukić, Arash Vahdat, Morteza Mardani
AffiliationsNVIDIA · Cornell University · Georgia Institute of Technology†Work done during internship at NVIDIA.*Core contributors, equal contribution.⋆Equal supervision.Correspondence to: mmardani@nvidia.com
Sigma brings continuous, steerable diffusion trajectories to large language models, aiming to make parallel generation faster and more controllable without sacrificing reasoning or coding ability.
Key results
Sigma diffuses over learned token embeddings in this-dimensional latent space.
Pass@1 in diffusion mode.
Pass@1 in diffusion mode.
Compared with 87.6% for Nemotron-Labs-Diffusion.
Percentage-point improvement in six-task average over the model without SUSReg.
Percentage-point improvement in six-task average over the model without SUSReg.
What the paper found
NVIDIA’s Sigma scales continuous diffusion language modeling to 3B and 8B parameters by denoising tokens in learned 16-dimensional embedding space, in parallel within blocks and conditioned on the preceding text. Likelihood-based training learns the embedding geometry, while warm-starting from autoregressive checkpoints and adding an auxiliary autoregressive objective stabilize training. At inference, classifier-free guidance and score temperature steer the denoising trajectory to balance answer quality and diversity; Sketched Uniform Spherical Regularization (SUSReg) and Parallel Decoding Distillation (PDD) help preserve performance with fewer steps. In diffusion mode, Sigma-3B scores 84.50% on GSM8K, and Sigma-8B scores 89.07%; the paper reports broadly competitive results against masked diffusion models including Nemotron-Labs-Diffusion, LLaDA, and Dream, as well as autoregressive models such as Llama-3.1 and Qwen3. On MBPP, Sigma reaches 91.8% pass@256, compared with 87.6% for Nemotron-Labs-Diffusion, illustrating its ability to find diverse correct solutions when sampling many responses. For few-step generation, combining SUSReg with PDD improves the six-task average by 3.71 percentage points at 4 NFEs and 4.67 percentage points at 8 NFEs over the corresponding model without SUSReg. The results position smooth, steerable continuous trajectories as a promising alternative to categorical diffusion for reasoning and code generation, while showing that very low-step decoding still loses accuracy.
Original abstract
Despite the success of discrete diffusion language models (dLMs) for fast parallel decoding, their non-smooth, high-dimensional space hinders trajectory steering for reasoning and inference acceleration. To overcome this, we present Sigma, the first large-scale (3B/8B) continuous dLM built on steerable, low-dimensional ODE/SDE latent trajectories. Trained blockwise via likelihood optimization, Sigma jointly denoises Gaussian-corrupted token embeddings while learning an optimal embedding geometry. To accelerate training, Sigma leverages pre-trained weights from autoregressive (AR) models for warm-starting. During inference, we identify classifier-free guidance and score temperature as essential for high-fidelity reasoning and coding. Across comprehensive math reasoning and coding evaluations against state-of-the-art discrete counterparts (masked dLMs and AR baselines), Sigma achieves competitive performance with discrete models on standard benchmarks (e.g., GSM8K, Minerva, HumanEval, MBPP) after pre-training and on challenging reasoning tasks (e.g., MATH-500, AIME) after supervised fine-tuning. Beyond performance parity, we uncover key structural properties unique to continuous dLMs: (i) embedding-space steering effectively governs the quality-diversity trade-off, yielding strong pass@k performance and (ii) continuous trajectories enable graceful degradation for low NFEs and efficient distillation. These establish continuous dLMs as a promising paradigm for efficient language generation.
Read the original paperMore in Diffusion Models
Browse all 67 papers →Does Physics Live in the Activations? Localizing Physical Quantities in Video Diffusion Models
Jonas Kneifl, Jakub Skalski, Bartłomiej Twardowski, Kamil Deja
This study looks inside video diffusion models to find where they represent physics—and shows those hidden representations can even be used to steer what the model generates.
Early Signatures of Memorization in Diffusion Models via Basin Geometry and Cyclic Denoising
Nikhil Verma, Siddharthan Dileep, Anoop Singh, Srikanth Sastry, Ramya Hebbalaguppe, Sayan Ranu, N. M. Anoop Krishnan
A new cyclic-denoising test can expose what a diffusion model has memorized before its usual outputs show any copies.
DuoMatching: Joint-Marginal Distribution Matching for Few-Step Video Generation
Jiahao Zhan, Yan Wang, Yongrui Ma, Qunliang Xing, Ruchang Yao, Runtao Liu, Shijie Zhao, Tianfan Xue
DuoMatching improves few-step video generation by combining video-level motion supervision with image-level visual and semantic guidance.