How Much Training-Inference Mismatch Can On-Policy Distillation Tolerate?
AuthorsCong Yu, Ziting Wang, Songlin Jiang, Sitong Zhang, Chengliang Chai, Youhe Jiang, Eiko Yoneki, Tuo Shi
AffiliationsNanyang Technological University · Independent · Beijing Institute of Technology · University of Cambridge · Shenzhen University of Advanced Technology
The study finds how much rollout mismatch language-model distillation can tolerate and shows how FP8 and asynchronous generation can make training faster without sacrificing quality.
Key results
nats per token for DeepSeek-R1-Distill-Qwen-7B; all tested configurations at or below this level passed.
nats per token; runs at this mismatch degraded, bounding the observed transition above 0.184.
More than 200 times the standard engine’s mismatch was tolerated by passing OPD configurations.
FP8 supported at least this many times as many KV-cache tokens as BF16.
FP8 with asynchronous rollouts reached synchronous BF16’s final held-out KL this many times sooner.
What the paper found
On-policy distillation trains a student on its own generated responses while a fixed teacher scores each token, but faster rollout systems can make the inference engine differ from the model being trained. This study measures that mismatch as per-token KL divergence and finds that DeepSeek-R1-Distill-Qwen-7B preserves training quality across a broad range: all tested configurations up to 0.184 nats per token passed, while runs at 0.39 nats per token degraded, placing the observed transition between those levels. The passing mismatch was more than 200 times the standard engine’s mismatch, whereas REINFORCE and GRPO broke at roughly 12 and 60 times that baseline. The safe range is model- and layer-specific: FP8 quantization of the first layer’s KV cache caused most of the mismatch in the DeepSeek model, while Llama-3.2-3B-Instruct was less sensitive. To make configuration choices practical, the paper introduces an admission check that estimates mismatch before a full training run. For efficiency, FP8 provides at least 1.93 times the KV-cache token capacity of BF16, and combining FP8 with asynchronous rollouts reached the synchronous BF16 run’s final held-out KL 1.62 times sooner. The results suggest OPD can exploit quantization and stale rollouts more safely than the tested RL methods, but only with a model-calibrated mismatch budget; per-token corrections do not address errors caused by the engine sampling different response prefixes.
Original abstract
On-policy distillation (OPD) trains a student language model on its own responses, guided by a fixed teacher. Generating these responses is expensive, so systems use quantized inference and asynchronous rollouts to reduce the cost. Both, however, introduce training-inference mismatch between the rollout engine and the model being trained. Such mismatch can destabilize reinforcement learning (RL), but its effects on OPD remain unclear. This paper investigates how much mismatch OPD tolerates, where its effects arise, and how training systems can use this tolerance. Measuring all mismatch sources on one scale, we examine quantization across three student-teacher pairs, with additional tests of weight staleness and injected noise on one pair. OPD tolerates substantial mismatch without meaningful loss of training quality but degrades at higher levels, revealing an empirical mismatch budget. Both the quality cost of mismatch and the mismatch introduced by quantization depend on the model, with some layers particularly sensitive to reduced precision. In controlled comparisons, OPD remains stable under rollout configurations that destabilize RL methods. Under weight noise, mismatch also shifts the prefixes reached during generation, introducing errors that per-token corrections cannot address. To put these findings into practice, we introduce a lightweight admission check that screens rollout configurations against a calibrated budget without requiring a full training run for each candidate. Model-specific FP8 recipes preserve training quality while providing capacity for at least $1.93\times$ as many KV-cache tokens as BF16. Combining FP8 with asynchronous rollouts reaches the same training quality $1.62\times$ sooner than synchronous BF16 training. Code is available at https://github.com/benyucong/train-inference-mismatch-opd
Read the original paperMore in Efficient AI
Browse all 79 papers →Latent Feedback Is a Compute and Data Amplifier
Benhao Huang, Chaitanya Dwivedi, Xuezhe Ma, Alexander Moreno, Hector Liu
Latent feedback may let language models get more value from each training token and FLOP than simply repeating the data.
MACaQ: Fast, Model Agnostic Calibrated Quantization for Low-Precision Weights and Activations
Lorenz K. Muller, Philippe Bich, Chiara Boretti, Hyun-Min Chang, Jiawei Zhuang, Lukas Cavigelli
MACaQ aims to make large neural networks cheaper to run by quantizing their weights and activations quickly, with little added inference overhead.
Mobile AI Factory: Building Specialized Language Models for Your Mobile Device
Evan Li, Krishna Deshpande, Wen Cheng, Shyamnath Gollakota
Mobile AI Factory automatically builds and tunes a small language model for a specific task and phone, aiming to deliver strong performance with less compute.