AttnSR: Attention Step Reward for Long-Horizon Agentic RL
AuthorsMengsong Wu, Tianchen Zhou, Zhaochen Su, Zhenhua Liu, Zhe Cui, Aimin Zhou
AffiliationsShanghai Institute of AI for Education, East China Normal University · Shanghai Innovation Institute · Hithink Research · School of Data Science and Engineering, East China Normal University · The Hong Kong University of Science and Technology · Nov 2026
AttnSR turns an LLM agent’s attention into step-level feedback to help it learn long-horizon tasks with fewer environment interactions.
Key results
Percentage-point increase from PPO to ATTN SR; final success rose from 91.24% to 97.08%.
Paired exact McNemar test for the ALFWorld success difference.
ALFWorld success, below ATTN SR’s 97.08%.
Reduction in mean environment steps versus PPO.
ATTN SR score; PPO scored 44.53%.
What the paper found
ATTN SR addresses long-horizon credit assignment by turning a policy’s own attention into intermediate rewards: during PPO replay, it traces attention links backward from the final step, combines layer-wise relevance with confidence gating, and assigns signed rewards based on whether the outcome beats a running baseline. The method retains the terminal reward and needs no extra rollouts or separately trained reward model; attention is treated as a relevance heuristic, not proof of causal contribution. With Qwen2.5-7B-Instruct, ALFWorld success rose from 91.24% with PPO to 97.08% with ATTN SR, a 5.84-point gain with paired p=0.002; a linear-step control reached 92.34%. On ScienceWorld, the final official score improved from 44.53% to 46.35%, alongside 29.88% fewer mean environment interactions. Results on WebShop were mixed: success edged up from 40.20% to 41.00%, but the official score fell from 70.74% to 69.71%. Overall, the strongest evidence is for improved ALFWorld performance, while gains do not generalize uniformly across benchmarks.
Original abstract
Long-horizon LLM agents make multi-step decisions, yet outcome-reward-only reinforcement learning provides little direct supervision for individual steps. We introduce AttnSR, a step-level reward augmentation method that derives reward allocations from policy attention extracted during training replay, without a separately trained reward model or additional rollouts. AttnSR traces backward through the policy model attention graph to estimate the relevance of earlier steps to the final response. The terminal step, measured relative to a running baseline, determines whether these rewards reinforce or penalize the selected steps. In a controlled Qwen2.5-7B-Instruct study, AttnSR consistently outperforms the PPO baseline across all evaluated checkpoints on ALFWorld, raising final held-out success from 91.2% to 97.1%. The ablation experiment shows that the benefit depends on how intermediate rewards are assigned, rather than simple accumulation of step rewards over time. AttnSR also improves ScienceWorld at every checkpoint while cutting environment interactions by 29.9%. These results support policy attention as a practical prior for step-level reward allocation in long-horizon agentic RL. Code is available at https://github.com/fairyshine/AttnSR .
Read the original paperMore in Reinforcement Learning
Browse all 69 papers →MAC-v2: Long-horizon model-based value expansion from pixels
Kwanyoung Park, Sergey Levine
MAC-v2 helps offline RL agents imagine further into the future from pixels by learning useful latent dynamics and predicting actions in chunks.
Training Stability under Adaptive Curricula
Johan Obando-Ceron, Walter Mayor, Michael Dennis, Aaron Courville, Pablo Samuel Castro
Adaptive training environments can undermine an RL agent’s ability to learn, so stable learners are essential for lasting performance.
On the Origin and Evolution of Training-Inference Mismatch in LLM Reinforcement Learning
Yu Li, Yufa Zhou, Tian Lan
The paper traces why LLM reinforcement learning can drift when sampling and training differ, then proposes a correction that detects when stale samples need resynchronization.