2607.01763

This paper revisits on-policy self-distillation for continual post-training and argues that denser token-level supervision is not a default stabilizer. Using Qwen3-4B-Thinking-2507 and Olmo-3-7B variants, the authors compare self-distillation policy optimization, or SDPO, against sequence-level GRPO across four domains: MATH, Science, Tool Use, and Coding. They find SDPO can strongly accelerate specialization on the current task—on MATH, SDPO0 raises AIME from 32.71% to 56.42% and Math500 from 73.85% to 84.29%—but the same dense supervision increases drift, interference, and collapse under domain shifts. A teacher-EMA sweep shows no globally optimal update rate, and a restart-and-freeze teacher variant repairs the worst case: at 5% EMA, the average score improves from 65.34% to 71.84%, and AIME jumps from 34.38% to 55.00%. However, in continual training over MATH → Science → Tool Use → Coding, GRPO retains capabilities far better, ending at 57.29% on AIME and 64.68% on Coding LCBv6, while SDPO0 falls to 34.38% and 61.77%, SDPOCoT0 collapses further, and SDPO5% drops near zero after the third stage. The paper attributes these failures to larger parameter and response drift, intermediate-distance interference, and a teacher-projection effect that can reinforce formatting artifacts through self-reinforcing confirmation bias.