Rationale-Guided Policy Optimization: Learning to Reason with Adaptive Rationale Scaffolding
AuthorsHoang Phan, Minh Pham, Chau Pham, Chinmay Hegde, Trung Le, Qi Lei
AffiliationsNew York University · University at Buffalo · Monash University
RGPO helps language models explore harder reasoning problems by temporarily guiding them with rationales, then feeding only their improved solutions back into reinforcement learning.
Key results
Qwen2.5-3B on DAPO-Math-27K; HINT scores 21.8.
Qwen2.5-3B; HINT scores 29.9.
RGPO result with Qwen2-VL-7B-Instruct.
RGPO overall score with Qwen2-VL-7B-Instruct.
RGPO reaches the GRPO baseline performance in 2.95× fewer steps on Qwen2-VL-2B.
Controlled simulation; GRPO reaches 0.0%.
What the paper found
With reasoning systems such as OpenAI’s GPT-6 Astra, Google DeepMind’s Gemini 4 Argon, and Anthropic’s Claude Opus 5.5 raising expectations, this paper addresses a basic training bottleneck: hard problems often produce no correct on-policy rollouts, leaving reinforcement learning with little useful feedback. Its method, Rationale-Guided Policy Optimization (RGPO), first lets a model attempt problems without hints, then adaptively reveals more of a reference rationale when the model fails and fades guidance when it succeeds. Guided attempts are kept only if a verifier scores them above the best unguided attempt; the accepted response is then distilled under the original prompt, so the model learns to solve without relying on the hint. On Qwen2.5-3B trained with DAPO-Math-27K, RGPO scores 26.8 in-distribution versus HINT’s 21.8, and 32.2 out-of-distribution versus 29.9. For vision-language reasoning, it reaches 70.1 on MMStar-Math and 73.65 on MathVista-Math with Qwen2-VL-7B-Instruct; on Qwen2-VL-2B, it matches the GRPO baseline in 2.95× fewer training steps. In a controlled simulation, RGPO reaches 99.5% hard-test accuracy, while GRPO remains at 0.0%. The results support the paper’s central distinction: rationales work best as temporary, verifier-checked search scaffolds, not fixed answers to imitate.
Original abstract
On-policy reinforcement learning has become a central paradigm for improving the reasoning abilities of large language models. However, its effectiveness is often limited by reward sparsity: when a model fails to discover correct trajectories for difficult problems, the optimization process receives little useful signal and may stagnate. Existing approaches mitigate this issue by incorporating off-policy demonstrations, expert traces, or model-generated solutions, but they typically require the auxiliary data to match the format of the reinforcement-learning task, often relying on rejection sampling from stronger models to obtain suitable training trajectories. We introduce Rationale-Guided Policy Optimization (RGPO), a framework that adaptively leverages ground-truth rationale information according to the model's current capability while preserving its freedom to explore. Rather than treating reference solutions as fixed imitation targets, RGPO uses them as temporary scaffolds: rationales help the model generate improved responses, after which only higher-reward, model-generated solutions are transferred back to the original unguided setting. This design allows training to exploit available ground-truth information without requiring off-policy data to follow the same format as the RL task. Across both language-only and vision-language reasoning settings, RGPO consistently improves performance over RLVR baselines, and ablation studies show that adaptive rationale guidance is a key contributor to these gains. These results suggest that RGPO offers a practical and general approach for reducing reward sparsity, stabilizing reinforcement learning, and improving reasoning performance in both text-only and multimodal models.
Read the original paperMore in Reinforcement Learning
Browse all 60 papers →On KL-Regularized Policy Optimization
Yifan Zhang
KLPO offers a cheaper, critic-free way to train asynchronous language-model agents from stale rollouts by using the sampler itself as the KL reference.
Sherpa: Teaching LLMs to Teach Adaptively
Weixian Xu, Yanzhe Zhang, Zora Zhiruo Wang, Changyu Chen, Diyi Yang
Sherpa trains LLM teachers through reinforcement learning with simulated students so they can adapt instruction and improve learner outcomes.
Gains and Collapse in On-Policy Distillation:A Reinforcement Learning Perspective
Han Cui, Jianhao Yan, Yun Luo, Hongbo Zhang, Zhizhang Fu, Yue Zhang
This paper explains why on-policy distillation can either improve language models or make them repetitive by amplifying behaviors that a teacher implicitly rewards.