Register Tokens for Bounded-State Reasoning in Diffusion Language Models

This paper introduces register tokens, a fixed-size continuous memory mechanism for masked diffusion language models, allowing systems such as LLaDA-8B-Base and Dream-7B-Base to continue reasoning after each generated text chunk is erased. Four fixed-position embeddings are written from a completed chunk, carried into the next window, and rewritten; training uses prompt-masked continuation passes and a fully masked pass so the next chunk must depend on the registers rather than reconstructing the solution from visible context. Post-training on a 60K-example mixture of OpenMathInstruct-2 and OpenCodeInstruct evaluates 128-token math chunks and 64-token code chunks. Registers outperform four-token discrete-text carry by up to 8.5 points on GSM8K and 19.5 points on MBPP, with Dream reaching 40.9 on MBPP versus 21.4 for discrete text. Their advantage is strongest when outputs span multiple chunks: on LLaDA, register carry achieves 26.2 on HumanEval and 29.2 on MBPP, exceeding full-sequence SFT by 12.2 and 3.5 points. Although full-context decoding remains more accurate at a 1024-token horizon, bounded registers reduce decoding cost, reaching a 5.6× speedup in the reported comparison. The proposed chunked diffu-GRPO reinforcement-learning method further improves register state quality, raising LongArithmetic reward by 8.1 points over discrete carry; linear probes also decode the running total with Pearson R=0.84 and the next operation with 80.0% accuracy. The results position registers as compact, writable reasoning state rather than simple token memory.