On KL-Regularized Policy Optimization
AuthorsYifan Zhang
AffiliationsOct 2026
KLPO offers a cheaper, critic-free way to train asynchronous language-model agents from stale rollouts by using the sampler itself as the KL reference.
Key results
Token-level KLPO can train from a single rollout per prompt.
What the paper found
KL-Regularized Policy Optimization, or KLPO, addresses asynchronous language-agent training, where stale checkpoints and differences between inference and training probabilities make rollout data off-policy. Instead of clipping importance ratios or sampling response groups as in GRPO, familiar from DeepSeekMath, KLPO anchors its KL penalty to the sampler and regresses the resulting Gibbs policy’s log-ratio condition on the sampler’s own trajectories. Profiling the regression intercept replaces an intractable normalizer with the sampler-mean signal plus the KL divergence from sampler to trainer. At the token level, the method derives critic-free gradients from terminal returns using sampler-centered scores or a trajectory residual; independent Monte Carlo estimates preserve unbiased gradients. Top-K and binary KL approximations are cheaper, but their exact error is the divergence within the merged tail. The framework identifies SPPO, GPO, REBEL, and BPO as special cases, and supports training with 1 rollout per prompt, without a critic, learned normalizer, or response group. The paper establishes these results theoretically and does not report benchmark scores.
Original abstract
Asynchronous reinforcement learning (RL) for large language model (LLM) agents trains one policy on trajectories generated by another: rollouts come from stale checkpoints, and the inference engine's probabilities differ from the trainer's even at identical parameters. Standard remedies either clip importance ratios, which biases the update, or, as in GRPO, sample a group of responses per prompt, which is costly when episodes are long. We propose KL-Regularized Policy Optimization (KLPO), a framework that anchors the KL regularizer at the sampler. The regularized improvement step then has a closed-form Gibbs solution, and KLPO fits its log-ratio optimality condition by least squares on the sampler's own trajectories, so the sampler probability enters through a log-ratio and no importance weights are needed. Profiling out the regression intercept replaces the intractable log-partition function with the signal's sampler mean plus a sampler-to-trainer KL divergence. For token-level policy mirror descent targets, we show that the resulting gradient can be computed from terminal returns without a critic, via sampler-centered scores or a single trajectory residual, even under stochastic tool outputs. We further prove that independent Monte Carlo estimates of the KL term keep these gradients unbiased, derive the exact KL gap of cheaper top-$K$ and binary approximations, and show that SPPO, GPO, REBEL, and BPO arise as special cases of KLPO. The result is a critic-free update that uses one rollout per prompt and requires neither a learned normalizer nor a group of responses.
Read the original paperMore in Reinforcement Learning
Browse all 60 papers →Rationale-Guided Policy Optimization: Learning to Reason with Adaptive Rationale Scaffolding
Hoang Phan, Minh Pham, Chau Pham, Chinmay Hegde, Trung Le, Qi Lei
RGPO helps language models explore harder reasoning problems by temporarily guiding them with rationales, then feeding only their improved solutions back into reinforcement learning.
Sherpa: Teaching LLMs to Teach Adaptively
Weixian Xu, Yanzhe Zhang, Zora Zhiruo Wang, Changyu Chen, Diyi Yang
Sherpa trains LLM teachers through reinforcement learning with simulated students so they can adapt instruction and improve learner outcomes.
Gains and Collapse in On-Policy Distillation:A Reinforcement Learning Perspective
Han Cui, Jianhao Yan, Yun Luo, Hongbo Zhang, Zhizhang Fu, Yue Zhang
This paper explains why on-policy distillation can either improve language models or make them repetitive by amplifying behaviors that a teacher implicitly rewards.