Gains and Collapse in On-Policy Distillation:A Reinforcement Learning Perspective
AuthorsHan Cui, Jianhao Yan, Yun Luo, Hongbo Zhang, Zhizhang Fu, Yue Zhang
AffiliationsZhejiang University · Westlake University
This paper explains why on-policy distillation can either improve language models or make them repetitive by amplifying behaviors that a teacher implicitly rewards.
Key results
Rate in the collapsed Qwen3-4B to Qwen3-1.7B-Base OPD setting.
Rate in the collapsed Qwen3-4B to Qwen3-1.7B-Base OPD setting.
Percentage-point improvement in average accuracy across four benchmarks for the Qwen3-4B setting.
Average accuracy across AMC23 and AIME24–26 after initializing from Qwen3-1.7B-SFT.
Percentage-point gain over the base-initialized Qwen3-4B OPD setting.
What the paper found
This paper reframes on-policy distillation, or OPD, as reinforcement learning: the teacher acts as an implicit reward model, scoring the student’s own generated responses through token-level reverse KL. Experiments with DeepSeek-R1-Distill-Qwen-1.5B and Qwen3 models on AIME24–26 and AMC23 show that OPD mainly makes correct answers the student could already produce easier to sample; it does not demonstrate expansion of the student’s problem-solving capability. In a targeted audit of 90 AIME problems, no problem remained solvable only by the distilled student after additional sampling and review. The same feedback mechanism can cause reward hacking when teacher preferences are misaligned with answer quality: with Qwen3-4B supervising Qwen3-1.7B-Base, 99.4% of trained responses hit the generation limit and 38% showed severe repetition, even though the teacher rarely generated such outputs. Masking truncated responses improved average accuracy by 3.08 percentage points, while initializing from Qwen3-1.7B-SFT reached 16.38% average accuracy, 6.22 percentage points above the base-initialized OPD setting. The central lesson is that OPD’s gains and failures depend not simply on how well a teacher generates, but on how reliably its feedback ranks the student’s rollouts.
Original abstract
On-policy distillation (OPD) has become an important approach to language model post-training. However, despite its performance gains, OPD can also collapse into excessively long and repetitive generation, and the mechanism underlying these divergent outcomes remains poorly understood. We explain these outcomes through a reinforcement learning perspective: the teacher implicitly rewards student behaviors, even those it rarely exhibits itself. From this perspective, our experiments show that OPD improves performance without expanding the student's capabilities. When the implicit reward model is reliable, OPD makes correct responses easier to sample. In contrast, when the preference misaligns with quality, reward hacking happens: the implicit reward model amplifies overlong, repetitive student rollouts, even though it rarely generates such text itself. Guided by this diagnosis, we find that masking unhealthy responses during training and using SFT initialization can each effectively mitigate the collapse. Together, these findings show that OPD amplifies student behaviors favored by the teacher's implicit feedback, shifting the focus from how well the teacher generates to how reliably it evaluates student rollouts. Our code is available at https://github.com/HancCui/opd_hacking.
Read the original paperMore in Reinforcement Learning
Browse all 57 papers →LoGRA: Scaling LLM Reinforcement Learning with Low-Rank Gradient Sketches
Shaokun Zhang, Yifan Zhang, Jian Hu, Yueying Li, Hao Zhang, Binfeng Xu, Jan Kautz, Yi Dong
LoGRA makes reinforcement learning for large language models more memory-efficient by compressing gradients and controlling update size.
Self-Retrospection Distillation: Turning Post-hoc Experiences into Prior Foresight
Haoxiang Zhang, Qinglin Chen, Hiroaki Hayashi, Zhuofeng Li, Siming Zhang, Jiaxin Zhang, Jixuan Chen, Fang Wu, Pan Lu, Silvio Savarese, Julian McAuley, Chien-Sheng Wu
SRD turns an agent’s hindsight about completed attempts into foresight that improves future decisions, especially when reinforcement-learning rewards provide little or no learning signal.
Res-HIL: Human-Guided Residual Reinforcement Learning for Sample-Efficient Dexterous Manipulation
Mariia Iavorskaia, Christian Dietz, Sebastian Albrecht, Majid Khadiv
Res-HIL lets humans efficiently improve robot manipulation skills by teaching a small corrective policy on top of an existing imitation policy.