Score Centering Stabilizes Off-policy Reinforcement Learning

This paper identifies training-inference mismatch as a central cause of instability in off-policy reinforcement learning for large language models, a problem relevant to systems behind models from OpenAI, DeepSeek, and Microsoft. The key mechanism is drift: when a quantized or stale sampler generates rollouts and a trainer computes gradients, the policy-gradient update contains a bias that repeatedly distills the trainer toward the sampler; synchronization then compounds the error until rewards collapse. The proposed solution, score centering, subtracts the sampler’s expected token score from each trainer score, canceling drift additively without importance ratios, clipping, masking, or a critic. Experiments spanning Qwen3 models from 0.6B to 30B parameters on Countdown and INTELLECT-2 show that score centering matches or beats truncated and masked importance sampling under severe quantization. With Qwen3-30B-A3B-Base on INTELLECT-2 math under FP8 weights and activations plus FP4 KV cache, score centering reaches 52% training accuracy versus 51% for truncated importance sampling; under INT8 weights and activations plus INT4 KV cache, it reaches 30% versus 12%. Under severe staleness, with the sampler updated every 64 steps, combining score centering with importance sampling performs best. For practical memory costs, the method logs only the sampler’s top 128 logprobs, matching full-vocabulary centering, with runs finishing within 1% of baseline wall-clock time.