Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding

NVIDIA’s Nemotron-Labs-Diffusion introduces a tri-mode language model that unifies autoregressive decoding, block-wise diffusion decoding, and self-speculation in one architecture, trained with a joint AR-plus-diffusion objective. The paper’s core claim is that AR and diffusion are complementary: adding the diffusion loss preserves or slightly improves AR accuracy, while the AR loss materially improves diffusion planning and stability. On 25B-token continuous pretraining, the ablation pipeline rises from 54.23% average accuracy with block-wise attention to 70.28% after adding global loss averaging, DP-rank varying masking ratios, two-stage training, and AR loss; the best diffusion weight is 𝛼=0.3. The released Nemotron-Labs-Diffusion-8B reaches 63.61% average accuracy in instruct mode, and its diffusion mode decodes 2.57 tokens per forward pass while its LoRA-tuned linear self-speculation reaches 5.99 tokens per forward. Against Qwen3-8B, the model’s AR mode is +0.86% better on average, its diffusion mode is +0.43% better, and on SPEED-Bench with SGLang on an NVIDIA GB200 GPU it delivers 4× higher throughput. A speed-of-light analysis shows a 7.60× diffusion upper bound on SPEED-Bench, 76.5% above linear self-speculation’s realized token-per-forward rate, indicating substantial remaining sampler headroom. The family scales to 3B, 8B, and 14B, extends to vision-language models, and uses a 20M-trajectory sampler trained with a 4-layer, 384-hidden-dimension Transformer.