On the Geometry of On-Policy Distillation

On the Geometry of On-Policy Distillation studies post-training dynamics in Qwen3 reasoning models and shows that on-policy distillation, or OPD, is neither supervised fine-tuning nor reinforcement learning with verifiable rewards, but a distinct geometry in parameter space. Using bf16-aware sparsity, principal-angle rotation, normalized spectral shift, and update-mask localization, the paper finds that OPD sits in a relaxed off-principal regime: in the controlled Qwen3-8B comparison, visible-update sparsity is 8.1% for SFT, 51.6% for OPD, and 77.2% for RLVR, while representative principal angles are above 10° for SFT, around 1° for OPD, and below 0.5° for RLVR. The authors then show subspace locking: OPD’s cumulative updates rapidly collapse into a narrow low-dimensional channel, and a rank-16 projection applied from about 20% of training preserves OPD performance but degrades SFT, indicating that the early subspace is functionally sufficient. Control experiments reveal that token sparsification at 25% or 50% and off-policy rollouts preserve this rank trajectory, but mixing OPD with RLVR changes it, so the critical lever is objective composition rather than runtime sampling. The study is framed around Qwen3-8B with a Qwen3-32B teacher on dapo-math-17k, and it argues for geometry-aware OPD designs that regulate update subspaces instead of only increasing token-level supervision density.