Experience-Handoff Reinforcement Learning for Recursive Self-Improvement
AuthorsDoyoung Kim, Byeongguk Jeon, Kailin Zhang, Sherry Yang
AffiliationsNew York University · KAIST AI
The work trains an AI agent not only to act, but also to write useful playbooks that help its future attempts succeed.
Key results
ACE-RL held-out score at equal training compute.
ACE-RL held-out score at equal training compute.
ACE-RL held-out score at equal training compute.
ACE-RL held-out score at equal training compute.
What the paper found
Experience-Handoff Reinforcement Learning addresses a gap between ordinary reinforcement learning, which trains an agent from task scores but discards detailed experience, and self-improvement harnesses, which preserve notes but do not train the model to write useful ones. The proposed ACE-RL method trains one model for three roles in the fixed Agentic Context Engineering harness: an actor that attempts the task, a reflector that extracts lessons from its trajectory, and a curator that edits a task-specific playbook. Its key technique is to test each candidate playbook in a fresh actor rollout and use the benchmark’s verifier reward as direct, verifiable feedback; group-relative comparisons within each role help assign credit without a learned reward model. At equal training compute, experiments with Alibaba Cloud’s Qwen3 models report ACE-RL RSI@3 scores of 0.775 on Mastermind, 0.987 on MBPP, 0.667 on AppWorld, and 0.340 on MiniSWEBench, outperforming the same-harness GRPO baseline on all four. The results support training not only for the current attempt, but also for the experience future attempts inherit.
Original abstract
It is common to train Large Language Models (LLMs) for agentic tasks with Reinforcement Learning (RL) from scalar rewards. However, this throws away the fine-grained information about the environment that the agent experienced during the trial (e.g., error logs). To utilize this rich information, recent works wrap the agent in a harness that refines its own inputs through reflection or compaction and hands past experience to future rollouts, but the construction of the handed-off artifacts (e.g., a playbook) is itself not learned. We want the best of both worlds: gradient updates from a scalar reward, while also utilizing environment information to hand off experience. As a concrete testbed, we fix the Agentic Context Engineering (\ace{}) harness and train one model to act, reflect, and curate task-local playbooks. Our method, \acerl{}, evaluates candidate experience-handoff artifacts by their downstream effect on future actor rollouts, and co-optimizes the role-specific behaviors (action generator, insight reflector, and playbook curator) under a fixed scaffold. Across agentic benchmarks, \acerl{} achieves higher held-out performance than baseline methods. The results suggest that agent training should optimize not only the actions in the current rollout, but also the model-written experience that future rollouts inherit.
Read the original paperMore in AI Agents
Browse all 77 papers →From Attack Surface to Defense Architecture: Compositional Security for Agentic Coding Harnesses
Simon Hegelich, Bardh Prenkaj, Habiba Sarhan, Morteza Shahrezaye, Kolja Hegelich, Christian Montag, Ralf Schneider, Erik Cambria
A security architecture aims to keep coding agents useful while preventing malicious instructions from turning their tools into an attack platform.
HarnessFlow: MCTS-Guided Harness Optimization for LLM Agents
Zhi Hong, Qianlong Wang, Hongchen Wei, Aidan Wu, Daohong Ying, Mingze Kong, Bei Liu, Chong Luo
HarnessFlow uses tree search and a coding agent to automatically refine the software harnesses that guide LLM agents.
Test-Time Compute Scales Better with Parallel Agent Teams
Junlin Wang, Haozhe Chen, Ang Li, Rahul Chalamala, Zhuang Liu
Instead of choosing between many solo agents and one huge team, run many small teams in parallel and keep the strongest result.