On the Generalization Limitation of On-Policy Distillation for Post-RL Consolidation
AuthorsYichen Wang, Yufa Zhou, Siqi Zhu, Haozheng Luo, Jingzhi Bao, Zhichao Liu, Zijun Wei
AffiliationsUniversity of Chicago · Duke University · UIUC · Nov 2026 · Northwestern University · ByteDance
Distilling a model’s skills can preserve benchmark performance while losing its ability to generalize, and this paper explores ways to prevent that.
Key results
Average shortfall versus the strongest teachers on composed tasks.
OPD update norm as a fraction of the RL update norm in matched math experiments.
What the paper found
On-policy distillation, or OPD, trains a model on its own generated responses using a teacher’s token-level feedback; multi-teacher OPD, or MOPD, extends this to several specialist teachers. In experiments with Qwen3 models, students reached 96.04–102.73% of their teachers’ in-distribution scores, yet their out-of-distribution results could remain 2.45–16.25% below the base model. Across composed tasks—such as coding with long-context reasoning—MOPD improved individual skills but averaged 23.82% below the strongest teachers on tasks requiring those skills together. The study also found that OPD’s parameter updates were smaller than reinforcement learning’s: in matched math experiments, their norm was 55.4% of the RL update norm. Two proposed fixes show promise: Procrustes-attention patching raised AIME25 from 60.83% to 65.00%, while replaying base-model behavior raised it to 67.50%, surpassing both the base and the RL teacher. Broader supervision also mattered: pairing Qwen3-32B with 10% OpenR1 data let the Qwen3-8B student exceed the base across all five math benchmarks. The results expose a blind spot in post-RL consolidation and are relevant to pipelines used by companies such as ByteDance: strong performance on supervised tasks does not guarantee that capabilities transfer or combine out of distribution.
Original abstract
Multi-teacher on-policy distillation (MOPD) extends on-policy distillation (OPD) to consolidate capabilities from multiple teachers. Although prior work reports near-perfect in-distribution (IND) distillation performance, we find that this can mask incomplete capability consolidation beyond the supervised distribution. A preliminary study on synthetic reasoning shows that OPD does not fully transfer teachers' generalization capabilities. We then extend this investigation to real-world tasks, evaluating extrapolative and compositional generalization across mathematical reasoning, coding, agent, and long-context capabilities using Qwen3 models. Across extrapolative settings, OPD and MOPD reach 96.04-102.73% of their teachers' IND scores. Yet their out-of-distribution (OOD) scores remain 2.45-16.25% below the base model's. Across three compositional settings, MOPD improves IND performance over the base model, but its performance on composed tasks as OOD remains 18.19-24.34% below that of the best teachers. To understand these gaps, we examine teacher capability and data coverage as two potential bottlenecks. Jointly alleviating these bottlenecks eliminates OOD forgetting in our experiments and suggests the potential for generalization beyond the teacher. (M)OPD also reduces OOD forgetting on average relative to its reinforcement learning (RL) baselines. Analysis of the associated update geometry reveals smaller and more spectrally concentrated updates on average than those of the RL baselines, along with greater shared update structure under MOPD. Motivated by these findings, we propose geometry-based patching and base-preserving replay to improve OOD performance while largely preserving IND performance. Our findings identify a blind spot in capability consolidation via (M)OPD, highlighting the need for algorithms that explicitly support OOD retention and generalization.
Read the original paperMore in Large Language Models
Browse all 99 papers →All Quiet on the Neural Front: Language Models Do Not Report Injected Pain States
Emilio Ferrara
Models’ internal signals can reveal an injected pain-like state and change their behavior, even when the models themselves do not report feeling it.
Knowing When to Teach: Learning from Teachable Prefixes in Multilingual Reasoning Distillation
Jaewon Cho, Minseo Yoon, Jinyoung Park, Jinwoo Seo, Jaewon Chu, Alice Oh, Hyunwoo J. Kim
The paper teaches multilingual reasoning models only on parts of their own reasoning where a teacher can still reliably guide them.
Cross-Primitive Consistency in Jev
Xu Liu, Guikun Chen, Kanzhi Wu, Wenguan Wang
PrimaCons measures when LLM decisions change across equivalent query formats, and CPRC helps make those decisions more consistent.