NTH

AttnSR: Attention Step Reward for Long-Horizon Agentic RL

AuthorsMengsong Wu, Tianchen Zhou, Zhaochen Su, Zhenhua Liu, Zhe Cui, Aimin Zhou

AffiliationsShanghai Institute of AI for Education, East China Normal University · Shanghai Innovation Institute · Hithink Research · School of Data Science and Engineering, East China Normal University · The Hong Kong University of Science and Technology · Nov 2026

November 16, 2026 2 min read
Watch on YouTube
The one-line take

AttnSR turns an LLM agent’s attention into step-level feedback to help it learn long-horizon tasks with fewer environment interactions.

Key results

5.84
ALFWorld success gain

Percentage-point increase from PPO to ATTN SR; final success rose from 91.24% to 97.08%.

0.002
ALFWorld paired p-value

Paired exact McNemar test for the ALFWorld success difference.

92.34%
Linear-step control success

ALFWorld success, below ATTN SR’s 97.08%.

29.88%
ScienceWorld interaction reduction

Reduction in mean environment steps versus PPO.

46.35%
ScienceWorld final official score

ATTN SR score; PPO scored 44.53%.

What the paper found

ATTN SR addresses long-horizon credit assignment by turning a policy’s own attention into intermediate rewards: during PPO replay, it traces attention links backward from the final step, combines layer-wise relevance with confidence gating, and assigns signed rewards based on whether the outcome beats a running baseline. The method retains the terminal reward and needs no extra rollouts or separately trained reward model; attention is treated as a relevance heuristic, not proof of causal contribution. With Qwen2.5-7B-Instruct, ALFWorld success rose from 91.24% with PPO to 97.08% with ATTN SR, a 5.84-point gain with paired p=0.002; a linear-step control reached 92.34%. On ScienceWorld, the final official score improved from 44.53% to 46.35%, alongside 29.88% fewer mean environment interactions. Results on WebShop were mixed: success edged up from 40.20% to 41.00%, but the official score fell from 70.74% to 69.71%. Overall, the strongest evidence is for improved ALFWorld performance, while gains do not generalize uniformly across benchmarks.

Original abstract

Long-horizon LLM agents make multi-step decisions, yet outcome-reward-only reinforcement learning provides little direct supervision for individual steps. We introduce AttnSR, a step-level reward augmentation method that derives reward allocations from policy attention extracted during training replay, without a separately trained reward model or additional rollouts. AttnSR traces backward through the policy model attention graph to estimate the relevance of earlier steps to the final response. The terminal step, measured relative to a running baseline, determines whether these rewards reinforce or penalize the selected steps. In a controlled Qwen2.5-7B-Instruct study, AttnSR consistently outperforms the PPO baseline across all evaluated checkpoints on ALFWorld, raising final held-out success from 91.2% to 97.1%. The ablation experiment shows that the benefit depends on how intermediate rewards are assigned, rather than simple accumulation of step rewards over time. AttnSR also improves ScienceWorld at every checkpoint while cutting environment interactions by 29.9%. These results support policy attention as a practical prior for step-level reward allocation in long-horizon agentic RL. Code is available at https://github.com/fairyshine/AttnSR .

Read the original paper

More in Reinforcement Learning

Browse all 69 papers →
02Rl

Training Stability under Adaptive Curricula

Johan Obando-Ceron, Walter Mayor, Michael Dennis, Aaron Courville, Pablo Samuel Castro

Adaptive training environments can undermine an RL agent’s ability to learn, so stable learners are essential for lasting performance.

Read analysis