Sherpa: Teaching LLMs to Teach Adaptively
AuthorsWeixian Xu, Yanzhe Zhang, Zora Zhiruo Wang, Changyu Chen, Diyi Yang
AffiliationsStanford University · Georgia Tech · Carnegie Mellon University
Sherpa trains LLM teachers through reinforcement learning with simulated students so they can adapt instruction and improve learner outcomes.
Key results
Percentage-point improvement reported across student archetypes.
Average pedagogy score, compared with 52.5% for the base model.
Comparison score on MathTutorBench.
Share of pairwise judgments preferring SHERPA, excluding ties.
What the paper found
SHERPA trains language models to adapt their teaching to a learner’s needs, rather than optimizing against a fixed rubric for good instruction. Using Qwen3-8B as the teacher and Qwen3-1.7B as simulated students on MATH problems, it runs multi-turn preparation, tutoring, and testing episodes. Student archetypes encode different preferences; an answer-leakage gate blocks revealing the final answer, while an adaptive gate accepts instruction only when it matches the student’s preference. The teacher is optimized with reinforcement learning to increase the student’s test accuracy, and turn-level masking concentrates credit on accepted teaching turns. The paper reports an average student-performance gain of 20.5 percentage points. On MathTutorBench, SHERPA’s pedagogy score is 79.2%, up from the base model’s 52.5% and above Gemini 3.8 Flash’s 72.4%. In human comparisons, teachers preferred SHERPA’s responses in 79.6% of judgments. These results suggest that learning from student outcomes can produce adaptable teaching that generalizes beyond the preferences used in training, though the study evaluates simulated student learning and human ratings of responses rather than learning gains in real students.
Original abstract
Large language models (LLMs) have become increasingly capable problem solvers, but being able to solve a problem is not the same as being able to teach it. Existing approaches to training LLMs as teachers rely on demonstrations, preference data, or predefined pedagogical criteria that specify what good teaching looks like. However, these signals are often not grounded in individual student learning outcomes, where effective teaching strategies can vary substantially across learners. To address this, we introduce Sherpa, a multi-turn reinforcement learning framework that instantiates multiple student archetypes with LLMs conditioned on distinct learning preferences and trains a teacher model to adapt its instruction by directly maximizing their learning outcomes. Teacher LLMs trained with Sherpa improve instructed students' performance across all archetypes by an average of 20.5 percentage points. Under MathTutorBench's evaluation, Sherpa raises the overall pedagogy score from 52.5% to 79.2%, indicating better teaching responses. Our human studies show that the trained teacher is preferred over the base model in 79.6% of pairwise comparisons. Together, Sherpa trains LLM teachers to adapt to diverse simulated students and become better aligned with human teachers, paving the road towards AI tutors teaching real students.
Read the original paperMore in Reinforcement Learning
Browse all 60 papers →On KL-Regularized Policy Optimization
Yifan Zhang
KLPO offers a cheaper, critic-free way to train asynchronous language-model agents from stale rollouts by using the sampler itself as the KL reference.
Rationale-Guided Policy Optimization: Learning to Reason with Adaptive Rationale Scaffolding
Hoang Phan, Minh Pham, Chau Pham, Chinmay Hegde, Trung Le, Qi Lei
RGPO helps language models explore harder reasoning problems by temporarily guiding them with rationales, then feeding only their improved solutions back into reinforcement learning.
Gains and Collapse in On-Policy Distillation:A Reinforcement Learning Perspective
Han Cui, Jianhao Yan, Yun Luo, Hongbo Zhang, Zhizhang Fu, Yue Zhang
This paper explains why on-policy distillation can either improve language models or make them repetitive by amplifying behaviors that a teacher implicitly rewards.