Trajectory-Refined Distillation
AuthorsLi Jiang, Haoran Xu, Yichuan Ding, Amy Zhang
This paper improves LLM distillation by fixing bad reasoning prefixes at the trajectory level, helping smaller models learn better from teacher-guided rollouts.
Key results
TRD under OPD improves AMOBench Pass@16 from 12.8 to 17.9.
TRD under OPD improves AMOBench Pass@16 from 23.1 to 35.9.
TRD under OPSD reaches 61.5 on AMOBench.
Raw rollout verifier pass rate before refinement in the Qwen3-8B trajectory analysis.
Refined trajectory verifier pass rate after TRD in the Qwen3-8B trajectory analysis.
Median raw rollout length in the Qwen3-8B training-corpus analysis.
What the paper found
Trajectory-Refined Distillation (TRD), from McGill University, Mila Quebec AI Institute, and UT Austin, argues that the main weakness of on-policy distillation in Qwen3-style post-training is not token-level loss noise but prefix failure: once a rollout enters a wrong reasoning path, dense per-token KL produces a bimodal teacher distribution and a fragmented gradient that cannot reconstruct the correction path. TRD fixes this by adding a trajectory-level refinement step, where a raw student rollout y_o is rewritten by the teacher into a refined trajectory y_r before standard distillation, preserving on-policy support while moving supervision onto the correction path. Across Qwen3-1.7B and Qwen3-4B-Instruct-2507, and across OPD and OPSD, TRD is consistently best or tied-best on seven of eight benchmarks in each OPD block, with especially strong gains on AMOBench: Pass@16 rises from 12.8 to 17.9 for Qwen3-1.7B and from 23.1 to 35.9 for Qwen3-4B-Instruct-2507, while the Qwen3-8B OPSD setup reaches 61.5 on AMOBench. The method also broadens reasoning coverage, reducing refined-corpus verifier failures from 65.8% on y_o to 81.4% on y_r for Qwen3-8B, compressing median trajectory length from 7.7K to 0.88K tokens, and cutting total wall-clock to 9:20 versus 9:40 for vanilla OPSD. The paper’s key claim is that trajectory-level correction, not loss clipping or top-K token filtering, is the right scale for fixing prefix failure.
Original abstract
On-policy distillation (OPD) has become a central post-training tool for large language models (LLMs), providing dense per-token teacher supervision along the student's own rollouts. In this work, we identify a common structural cause underlying OPD, which we call prefix failure. Under prefix failure, dense per-token supervision induces a bimodal teacher mixture and fragmented gradients that token-level loss truncation or reweighting fail to address. This observation motivates us to move beyond token-level loss interventions toward trajectory-level output corrections. We thus propose Trajectory-Refined Distillation (TRD), a trajectory-level correction method that revises the student's rollout under the teacher guidance while within on-policy support. By correcting problematic prefixes before distillation, TRD mitigates prefix failure at its source. Moreover, TRD improves the exploration by exposing the student to alternative valid derivations under teacher guidance, even when the original rolls are already correct. TRD can also be applied to on-policy self-distillation (OPSD), a parameter-sharing variant that uses the student model conditioned on privileged informations as the teacher. Across a wide range of benchmarks and base models at multiple scales, TRD consistently outperforms prior baselines, improving single-attempt accuracy and broadening reasoning coverage. Code is available at https://github.com/louieworth/trd
Read the original paperMore in Large Language Models
Browse all 81 papers →Finetuning with Sampling: SFT Learns Better Than You Think
Aayush Karan, Sitan Chen, Yilun Du
By sampling and reshaping expert data before training, this work argues that supervised finetuning can match RL while generalizing better and forgetting less.
Generalization Dynamics of LM Pre-training
Jiaxin Wen, Zhengxuan Wu, Dawn Song, Lijie Chen
Language models may repeatedly switch between shallow memorization and genuine reasoning during training, and the paper shows how to detect and potentially control these swings.
Rethinking Self-Distillation for Multi-Teacher Capability Merging
Roy Xie, Dan Friedman, Feng Nan, Yukun Huang, Zhichao Xu, Chengjiu Zhang, Jun Xu, Manaal Faruqui, Vivek Rathod, Bhuwan Dhingra
The study finds that expensive multi-teacher on-policy distillation may offer little advantage over carefully tuned, cheaper alternatives such as SFT and weight merging.