What Outcome RL Buys: Finishing, Reliability, Rarely Reach
AuthorsShan Yang
AffiliationsIndividual Researcher
Outcome RL often helps models finish on time and answer reliably, but appears to add few genuinely new problem-solving capabilities.
Key results
Models compared with their thinking-model bases.
Share of pass@1 change attributed to finishing.
Upper bound across four physics and math test sets.
Five of 144 physics wins remained genuine after checks.
Rate rose from 0.10 on selected training problems; held-out transfer was not shown.
What the paper found
The paper decomposes pass@1 gains from outcome reinforcement learning into finishing within a token budget, reliability once a derivation finishes, and reach on problems the base model repeatedly misses. Across 20 released models derived from DeepSeek-R1 distills and Qwen3, tested on physics and math including MATH-500, finishing explains 82% of the change at 4k and 8k tokens; once the base usually finishes, reliability is the main gain. The authors’ reach measure finds little evidence of new problem-solving knowledge: net reach is at most 2.1 percentage points across four test sets, and only 3.5% of 144 audited RL-only physics wins held up as genuine new solutions. Trace checks, including readings assisted by Claude Opus 5.5, suggest RL more often makes models use steps already present in their base model’s samples. In a controlled GSPO experiment with Qwen3-VL-8B-Thinking, supplying physics setups did not show clear transfer to held-out problems; revisiting selected training problems raised their solve rate from 0.10 to 0.28, but did not improve held-out performance. The practical implication is to report finishing rates alongside pass@1 and compare models at or above their training token cap, so shorter answers are not mistaken for better reasoning.
Original abstract
Outcome RL raises pass@1 but not pass@k at large k (Yue et al., 2025). But every pass@1 is scored under a token budget. Every recipe also trains under one. On physics and math problems at fixed token budgets, we compare the pass@1 of 20 released RL models (1.5B to 32B) with that of the thinking models they started from. We ask whether each pass@1 gain is changed behaviour or new knowledge. We divide each change in pass@1 into three parts. Finishing measures answering before the budget runs out. Reliability measures being right once finished. Reach measures problems RL solves that its base answered wrongly in all eight tries, minus problems the base solves that RL answered wrongly in all eight. We find four things. (1) Where the base often hits the token budget before answering, most of the change, gain or loss, is finishing (82% at 4k and 8k). Given more tokens, the base closes most of the finishing gap itself. (2) Where the base has enough tokens, the gain is reliability: the RL model is graded right more often, even on problems both finish. (3) By our definition, RL rarely solves what its base cannot. Reach adds at most 2.1 percentage points on any of our four physics and math sets. We checked 144 physics problems that only the RL model solved. Of these, 3.5% hold up when resampled or read. In a random sample, nine of ten new RL steps a human confirmed also appear in the base's samples. (4) In a small-scale RL run on an 8B model, the gain on new problems is finishing, with reliability flat. Supplying each problem's physical setup was not shown to improve on plain RL.
Read the original paperMore in Reinforcement Learning
Browse all 66 papers →On the Origin and Evolution of Training-Inference Mismatch in LLM Reinforcement Learning
Yu Li, Yufa Zhou, Tian Lan
The paper traces why LLM reinforcement learning can drift when sampling and training differ, then proposes a correction that detects when stale samples need resynchronization.
ProAS: A Progress-Aware Advantage Shaping Framework for Multi-Turn Agentic Reinforcement Learning
Bowen Ding, Mingda Wan, Yuhan Chen, Jiyao Yuan, Qi Zhu, Futing Wang, Heyuan Deng, Fei Mi, Lifeng Shang, Tao Lin
ProAS helps language-model agents learn from multi-step tasks by assigning credit more selectively to the actions that make progress.
Enabling Token-wise Importance Sampling in Off-Policy RLVR with Proximal KL Regularization
Hao Bai, Jiarui Yao, Tong Zhang
A new proximal-KL approach makes token-wise off-policy learning more stable for training language models with verifiable rewards.