Self-Retrospection Distillation: Turning Post-hoc Experiences into Prior Foresight
AuthorsHaoxiang Zhang, Qinglin Chen, Hiroaki Hayashi, Zhuofeng Li, Siming Zhang, Jiaxin Zhang, Jixuan Chen, Fang Wu, Pan Lu, Silvio Savarese, Julian McAuley, Chien-Sheng Wu
AffiliationsSalesforce AI Research · UC San Diego · Texas A&M University · Stanford University
SRD turns an agent’s hindsight about completed attempts into foresight that improves future decisions, especially when reinforcement-learning rewards provide little or no learning signal.
Key results
Tool-use reasoning and agentic benchmark tasks.
Percentage-point gain from SRD over a baseline.
Share of groups in the 2B code-task stress test.
Baseline result in the 2B stress test.
Result under the same rollout budget in the 2B stress test.
What the paper found
Self-Retrospection Distillation, or SRD, turns an agent’s completed interactions into foresight targets: before acting, the policy predicts likely pitfalls or useful knowledge; afterward, a stopped-gradient EMA teacher, given the trajectory and its outcome, teaches those predictions through token-level generalized Jensen–Shannon divergence. This prospective loss complements GRPO and self-distillation, and foresight is only a training target, so the agent need not generate it at inference. Across 10 tool-use and agentic benchmarks, experiments with Qwen3.5-4B and Qwen3.5-9B report gains of up to 24.2 percentage points. The key advantage appears when reward-based learning has no contrast to use: in a 2B code-task stress test, 98% of rollout groups were reward-uniform, and GRPO finished at 0.0% training success, while GRPO plus SRD reached 60.6% with the same rollout budget. Pitfall supervision was generally the more efficient choice; adding a separate knowledge channel often brought little benefit. OpenAI’s gpt-5.6-luna was used as a fallback judge for some benchmark answers, not as the trained agent. The result is a way to learn from failures and successes that ordinary group-relative rewards discard, by teaching agents what their own experience could have helped them anticipate.
Original abstract
Reinforcement learning with verifiable rewards (RLVR) turns agent experience into learning signals primarily through scalar outcome rewards after interaction. For group-relative objectives, however, this signal vanishes when all rollouts receive the same reward, even though their trajectories may reveal useful information about what the task requires and how the agent fails. We ask a complementary question: can hindsight teach an agent what it could have anticipated before acting? We introduce prospective learning, which uses post-hoc experience to supervise foresight predictions from the pre-interaction view, and instantiate it with Self-Retrospection Distillation (SRD). Intuitively, a completed trajectory reveals knowledge that would have been useful and pitfalls that should be avoided; SRD distills this privileged hindsight into trajectory-blind foresight of the same policy. Foresight serves only as a training target and need not be explicitly generated at inference time. Across 10 tool-integrated reasoning and long-horizon agentic tasks, SRD complements RLVR and self-distillation baselines with gains of up to $24.2$ pp. Its advantage is especially pronounced when reward contrast is scarce: when $37$--$98\%$ of rollout groups are reward-uniform across model scales, yet SRD can still exploit learning signal from sampled trajectories. In the 2B setting, where $98\%$ of groups are all-failure, the RLVR training ends up at $0.0\%$ success, while adding SRD reaches $60.6\%$ under the same rollout budget. Our results suggest that post-hoc agent experience is useful not only for evaluating or improving behavior, but also for shaping predictive representations before available interaction.
Read the original paperMore in Reinforcement Learning
Browse all 57 papers →Gains and Collapse in On-Policy Distillation:A Reinforcement Learning Perspective
Han Cui, Jianhao Yan, Yun Luo, Hongbo Zhang, Zhizhang Fu, Yue Zhang
This paper explains why on-policy distillation can either improve language models or make them repetitive by amplifying behaviors that a teacher implicitly rewards.
LoGRA: Scaling LLM Reinforcement Learning with Low-Rank Gradient Sketches
Shaokun Zhang, Yifan Zhang, Jian Hu, Yueying Li, Hao Zhang, Binfeng Xu, Jan Kautz, Yi Dong
LoGRA makes reinforcement learning for large language models more memory-efficient by compressing gradients and controlling update size.
Res-HIL: Human-Guided Residual Reinforcement Learning for Sample-Efficient Dexterous Manipulation
Mariia Iavorskaia, Christian Dietz, Sebastian Albrecht, Majid Khadiv
Res-HIL lets humans efficiently improve robot manipulation skills by teaching a small corrective policy on top of an existing imitation policy.