Simplex Diffusion Models

Simplex Diffusion Models address information collapse in discrete diffusion by representing every intermediate token state as a probability vector on the simplex, rather than repeatedly sampling a single category. A Dirichlet forward process preserves uncertainty, while a closed-form reverse bridge enables a DDIM-like sampler with tunable stochasticity through the churn parameter, avoiding the ordinary differential equation integration required by Dirichlet Flow Matching. Training uses a standard cross-entropy objective, and the framework connects discrete and continuous diffusion: high temperature approaches categorical diffusion, while low temperature approaches deterministic interpolation with Gaussian fluctuations. On TinyGSM, an 11.8M-example synthetic math dataset generated with GPT-3.5 and evaluated on GSM8K using the SmolLM tokenizer, SDMs reached 49.0% accuracy at 512 steps without Self-Conditioning, versus 45.8% for masked diffusion with Self-Conditioning. Distillation reduced generation to 8 steps while solving 32.1% of GSM8K problems, exceeding distilled discrete diffusion’s 21.4% at 128 steps. On Sudoku, SDMs with Self-Conditioning achieved 99.1% accuracy, and on OpenWebText they reached 17.0 GenPPL at 5.46 unigram entropy under GPT-2-large evaluation, close to real validation statistics. The results suggest that continuous belief states can improve discrete generation, although experiments remain limited to relatively small models.