NTH

Every Token Needs Its Own Teacher: Multi-teacher Adaptive On-policy Self-distillation

AuthorsYifu Luo, Chuang Ma, Zeyu Chen, Enkelejda Kasneci, Qi Chen, Gjergji Kasneci

AffiliationsTU München · Munich Center for Machine Learning (MCML) · relAI – Konrad Zuse School of Excellence in Reliable AI · Kyoto University · NII LLMC · ELLIS Institute Finland · Aalto University

November 9, 2026 2 min read
Watch on YouTube
The one-line take

The method lets each token in an LLM response learn from whichever candidate teacher best suits it.

Key results

48.44%
Preferred-teacher overlap at 25% and 50% rollout positions

Pilot study shows teacher preference can differ between tokens in one response.

5.9
Qwen3-1.7B average-score improvement

Points gained over the base model across AMC23, AIME24, and AIME25.

1.5
Qwen3-4B average-score improvement

Points gained over the base model across AMC23, AIME24, and AIME25.

2.8
Qwen3-8B average-score improvement

Points gained over the base model across AMC23, AIME24, and AIME25.

What the paper found

Multi-teacher Adaptive On-policy Self-distillation, or MTA-OPSD, argues that one response-wide teacher cannot supervise every token effectively: the best teacher can change from token to token. In a teacher-intervention study using Qwen3, preferred-teacher overlap between tokens at the 25% and 50% rollout positions was only 48.44%. The method builds a pool of seven teachers by varying how much of a reference solution is exposed and which portion is retained; for each token, majority voting filters teachers by whether they support the student’s sampled token, then KL divergence selects the most distinct teacher in that agreeing group. Tested on mathematical reasoning with DeepMath-103K training data and AMC23, AIME24, and AIME25 evaluations, MTA-OPSD improved average benchmark scores over the base model by 5.9 points for Qwen3-1.7B, 1.5 for Qwen3-4B, and 2.8 for Qwen3-8B. It also outperformed fixed-teacher OPSD, SDPO, and the adaptive-reference method Paint, indicating that token-level teacher assignment—not merely using multiple teachers—is central to the gains.

Original abstract

On-policy Self-distillation (OPSD) has proven effective for post-training large language models by constructing a teacher conditioned on privileged references. However, existing approaches keep the teacher fixed either throughout training by consistently exposing it to the full reference, or within each response by adaptively varying partial reference exposure only across responses. In this paper, we argue that neither of them is sufficient. Instead, every token needs its own teacher, as the teacher preference can vary token by token within the same response. Through a teacher-intervention pilot study, we validate that the teacher preference differs across tokens, and reveal that a preferred teacher can be identified for each token through majority voting and KL-based selection. Motivated by these findings, we propose Multi-teacher Adaptive On-policy Self-Distillation (MTA-OPSD), the first token-level adaptive OPSD method. Our method constructs a multi-teacher pool consisting of candidate teachers, and enables every token to adaptively select its preferred teacher during training. Experiments with Qwen3 models on mathematical reasoning tasks show that MTA-OPSD consistently outperforms existing OPSD approaches with an average gain of up to $+5.9$ over the base model, highlighting the importance of adapting teacher supervision at the token level. The code is available at https://github.com/xingzhejun/MTA-OPSD.

Read the original paper

More in Large Language Models

Browse all 99 papers →