NTH

Experience-Handoff Reinforcement Learning for Recursive Self-Improvement

AuthorsDoyoung Kim, Byeongguk Jeon, Kailin Zhang, Sherry Yang

AffiliationsNew York University · KAIST AI

November 16, 2026 2 min read
Watch on YouTube
The one-line take

The work trains an AI agent not only to act, but also to write useful playbooks that help its future attempts succeed.

Key results

0.775
Mastermind RSI@3

ACE-RL held-out score at equal training compute.

0.987
MBPP RSI@3

ACE-RL held-out score at equal training compute.

0.667
AppWorld RSI@3

ACE-RL held-out score at equal training compute.

0.340
MiniSWEBench RSI@3

ACE-RL held-out score at equal training compute.

What the paper found

Experience-Handoff Reinforcement Learning addresses a gap between ordinary reinforcement learning, which trains an agent from task scores but discards detailed experience, and self-improvement harnesses, which preserve notes but do not train the model to write useful ones. The proposed ACE-RL method trains one model for three roles in the fixed Agentic Context Engineering harness: an actor that attempts the task, a reflector that extracts lessons from its trajectory, and a curator that edits a task-specific playbook. Its key technique is to test each candidate playbook in a fresh actor rollout and use the benchmark’s verifier reward as direct, verifiable feedback; group-relative comparisons within each role help assign credit without a learned reward model. At equal training compute, experiments with Alibaba Cloud’s Qwen3 models report ACE-RL RSI@3 scores of 0.775 on Mastermind, 0.987 on MBPP, 0.667 on AppWorld, and 0.340 on MiniSWEBench, outperforming the same-harness GRPO baseline on all four. The results support training not only for the current attempt, but also for the experience future attempts inherit.

Original abstract

It is common to train Large Language Models (LLMs) for agentic tasks with Reinforcement Learning (RL) from scalar rewards. However, this throws away the fine-grained information about the environment that the agent experienced during the trial (e.g., error logs). To utilize this rich information, recent works wrap the agent in a harness that refines its own inputs through reflection or compaction and hands past experience to future rollouts, but the construction of the handed-off artifacts (e.g., a playbook) is itself not learned. We want the best of both worlds: gradient updates from a scalar reward, while also utilizing environment information to hand off experience. As a concrete testbed, we fix the Agentic Context Engineering (\ace{}) harness and train one model to act, reflect, and curate task-local playbooks. Our method, \acerl{}, evaluates candidate experience-handoff artifacts by their downstream effect on future actor rollouts, and co-optimizes the role-specific behaviors (action generator, insight reflector, and playbook curator) under a fixed scaffold. Across agentic benchmarks, \acerl{} achieves higher held-out performance than baseline methods. The results suggest that agent training should optimize not only the actions in the current rollout, but also the model-written experience that future rollouts inherit.

Read the original paper

More in AI Agents

Browse all 77 papers →