MagicBeanStalk: Escaping the RLVR Zero Advantage Trap with Recursive Data Synthesis
AuthorsXiming Lu, Aviral Kumar, Jan Kautz, Yejin Choi, David Acuna
AffiliationsUniversity of Washington · Carnegie Mellon University · NVIDIA · Stanford University
MagicBeanStalk recursively creates calibrated reasoning problems that help RL-trained models learn more effectively than simply scaling or filtering existing data.
Key results
Approximate share of the math corpus yielding all-success or all-failure rollout groups.
Average-point improvement across 8 math benchmarks after continual RLVR.
Average-point improvement across 8 math benchmarks using the same synthetic data.
More than this number of verifiable problems were synthesized from 340 expert-curated questions.
Point improvement for Qwen3-4B-Instruct after RLVR training on synthesized finance data.
Point improvement for Qwen3-4B-Instruct after RLVR training on synthesized finance data.
What the paper found
MagicBeanStalk tackles the RLVR zero-advantage trap: when every sampled response to a problem is either right or wrong, group-relative advantages are zero, and roughly 88% of a large math corpus supplied no such learning signal for the tested policy and rollout budget. Instead of merely filtering those problems or asking a teacher for surface-level variants, MagicBeanStalk decomposes verified hard problems into easier subproblems, synthesizes harder new leaves, and recursively recombines them while preserving the original reasoning structure. It keeps problems only when the target policy produces mixed correct and incorrect rollouts and at least one complete correct solution within budget. Continual RLVR on the resulting data raises gpt-oss-120b’s average across 8 math benchmarks from 61.2 to 69.8, a gain of 8.6 points; the same data raises DeepSeek v4 Flash from 72.4 to 77.9, a gain of 5.4 points. The method also turns 340 expert-curated finance questions into more than 14K verifiable problems. Training Qwen3-4B-Instruct on these improves pass@1 by 6.9 points on SecQue and 2.4 points on FinanceBench. The central contribution is a recursive synthesis process that uses hard problems as specifications of missing reasoning skills, then calibrates new training problems to both the model and its response budget.
Original abstract
Data-centric scaling of RLVR has focused on collecting or generating verifiable training problems. However, corpus size alone does not determine whether the dataset provides a useful learning signal for a target policy. To guide the synthesis of effective RLVR data, we experimentally characterize dataset effectiveness as a function of its problems, the target policy, and the training rollout budget. Under this characterization, we show that a large open-source math corpus pooled from widely used RLVR datasets falls almost entirely into a zero-advantage trap: for a fixed target policy and rollout budget, roughly 88% of its problems are solved in every rollout or failed in every rollout. Filtering concentrates what remains, but it can only select, discarding the hard problems. Effective synthetic problems should therefore require the reasoning operations and compositions required to solve those hard problems. Yet their difficulty must be calibrated, so that some rollouts succeed, some fail, and at least one correct solution fits within the training budget. Prompting a teacher to vary hard problems directly does not achieve this. Guided by these empirical principles, we introduce MagicBeanStalk, a top-down decomposition and bottom-up recursive data-synthesis algorithm that grows seed problems into families of substantively new, policy- and budget-calibrated training problems while preserving the seeds' reasoning blueprints. Across 8 math benchmarks, continual RLVR on MagicBeanStalk problems improves gpt-oss-120b from 61.2 to 69.8 (+8.6) and DeepSeek v4Flash from 72.4 to 77.9 (+5.4) on a 16K-token budget. Training on the full 78K source corpus improves by only +1.1 and its best filtered real data subset by +5.8. In finance, we synthesize over 14K verifiable problems from only 340 solutions and improve over base by 6.9 and 2.4 points on SecQue and FinanceBench, respectively.
Read the original paperMore in Reinforcement Learning
Browse all 63 papers →Enabling Token-wise Importance Sampling in Off-Policy RLVR with Proximal KL Regularization
Hao Bai, Jiarui Yao, Tong Zhang
A new proximal-KL approach makes token-wise off-policy learning more stable for training language models with verifiable rewards.
LLM Value Functions for Credit Assignment in Agentic RL
Kaijie Zhu, Weixiang Yan, Omar Attia, Abbas Kazerouni, Alexander Toshev, Andrew Szot
LVRL uses an LLM’s value estimates to identify which actions help or hurt an agent, improving learning from sparse rewards.
On KL-Regularized Policy Optimization
Yifan Zhang
KLPO offers a cheaper, critic-free way to train asynchronous language-model agents from stale rollouts by using the sampler itself as the KL reference.