NTH
Reinforcement Learning

ProAS: A Progress-Aware Advantage Shaping Framework for Multi-Turn Agentic Reinforcement Learning

AuthorsBowen Ding, Mingda Wan, Yuhan Chen, Jiyao Yuan, Qi Zhu, Futing Wang, Heyuan Deng, Fei Mi, Lifeng Shang, Tao Lin

AffiliationsZhejiang University · School of Engineering, Westlake University · Huawei Noah’s Ark Lab 2 25 Oct 2026 4 · Institute of Advanced Technology, Westlake Institute for Advanced Study

November 3, 2026 2 min read
Watch on YouTube

Resources

View on Hugging Face Read the original paper on arXiv GitHub repository
The one-line take

ProAS helps language-model agents learn from multi-step tasks by assigning credit more selectively to the actions that make progress.

Key results

94.3%
ALFWorld success rate

Qwen2.5-7B-Instruct with GiGPO+ProAS.

70.8%
WebShop success rate

Qwen2.5-7B-Instruct with GRPO+ProAS.

2.44
Maximum convergence speedup

Maximum training convergence speedup reported with ProAS.

16.7
ALFWorld GRPO success-rate gain

Percentage-point gain with the 1.5B model after adding ProAS.

15.4
ALFWorld gain over GPT-5.6-Sol

Percentage-point advantage for Qwen2.5-7B-Instruct with GiGPO+ProAS.

What the paper found

ProAS addresses a credit-assignment weakness in critic-free multi-turn agent reinforcement learning: methods such as GRPO apply one trajectory-level advantage uniformly, potentially reinforcing bad actions in successful rollouts and penalizing early decisions no more than later failures. ProAS uses a dynamically updated Agentic Reference Graph to label actions as reasonable and progress-advancing, then applies Masking Unreasonable Incentives to avoid reinforcing invalid actions and Asymmetric Penalty Redistribution to weight negative updates by each action’s downstream Future-Drift. This lightweight, critic-free framework plugs into GRPO, DAPO, and GiGPO. With Qwen2.5-7B-Instruct, GiGPO+ProAS reaches 94.3% success on ALFWorld, 15.4 percentage points above OpenAI’s GPT-5.6-Sol, while GRPO+ProAS reaches 70.8% on WebShop. ProAS speeds convergence by up to 2.44×; on ALFWorld, adding it to GRPO improves success by 16.7 percentage points with the 1.5B model. Results on ALFWorld and WebShop, plus tests with incomplete and teacher-distilled reference graphs, suggest the approach can improve performance without requiring a learned critic.

Original abstract

Reinforcement learning (RL) has emerged as the primary paradigm for optimizing Large Language Model (LLM) agents, yet efficient critic-free algorithms (e.g., GRPO) suffer from a fundamental uniform credit assignment dilemma that broadcasts trajectory-level advantages equally to all actions. Our analysis reveals that this uniformity inherently reinforces structurally unreasonable actions in preferred trajectories and ignores each action's downstream deviations in unpreferred ones. To resolve this without introducing critic overhead, we propose Progress-Aware Advantage Shaping (ProAS), a plug-and-play framework that leverages a dynamically evolving Agentic Reference Graph (ARG) to achieve fine-grained credit assignment via Masking Unreasonable Incentives (MUI) and Asymmetric Penalty Redistribution (APR). Extensive experiments across Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct demonstrate that ProAS universally elevates mainstream group-based baselines to new state-of-the-art performance, achieving peak success rates of 91.9% and 94.3% on ALFWorld, alongside 66.7% and 70.8% on WebShop, respectively. Furthermore, it accelerates training convergence by up to $2.44\times$ while maintaining strong robustness against imperfect or distilled ARGs. Codes: https://github.com/LINs-lab/ProAS

Read the original paper

More in Reinforcement Learning

Browse all 66 papers →