Think Less, Locally: Local Preference Learning for Reasoning Compression
AuthorsMuyao Yuan, Liyan Xu, Weizhan Zhang, Muyan Jiao, Jiangnan Li, Jinchao Zhang, Jie Zhou
AffiliationsMOEKLINNS, School of Computer Science and Technology, Xi’an Jiaotong University · WeChat AI, Tencent Inc
LoCo teaches reasoning models when to stop thinking, cutting response length while preserving or improving accuracy.
Key results
Reduction relative to the base model.
LoCo accuracy; the base model scores 50.83%.
Reduction relative to the base model after math-only training.
Reduction relative to the base model after math-only training.
What the paper found
LoCo teaches a reasoning model when to stop, rather than simply rewarding shorter answers. It detects likely overthinking states in the model’s own traces using low confidence and high confidence volatility, then asks a Qwen3-30B judge to confirm redundancy. From each confirmed state, it freezes the prefix, generates eight continuations, selects the shortest one verified against the answer, and uses the original continuation as the rejected option in DPO training. Trained on math-only traces from DeepSeek-R1-Distill-Qwen-7B, LoCo reduces completion length while preserving or improving accuracy across five math and three code benchmarks, and achieves the best AUC on all three code benchmarks. On AIME24, it cuts completion length by 45.4% and reaches 54.17% accuracy, compared with 50.83% for the base model. Despite training only on math, it reduces tokens by 36.3% on MBPP and 23.4% on CodeContests. Ablations, including one using GPT-5.4-generated continuations, support the value of learning local preferences from the model’s own reasoning states.
Original abstract
Large reasoning models achieve strong performance on challenging tasks, but their long chain-of-thought often incurs substantial inference costs by continuing to deliberate after sufficient information has been obtained. Existing reasoning compression methods primarily operate on reasoning trajectories, paying less attention to the local decision of whether further reasoning is necessary given the current state. We propose LoCo, a local preference learning method for reasoning compression. LoCo detects candidate overthinking states from the model's own traces, freezes the corresponding reasoning prefixes, regenerates shorter verified continuations from the same states, and trains on the resulting state-conditioned preference pairs with DPO. This formulation teaches the model when to continue reasoning and when to conclude, rather than simply encouraging shorter trajectories overall. We construct training data entirely from self-generated mathematical traces and evaluate LoCo on five mathematical and three code generation benchmarks. LoCo reduces completion length by 41.7\%--45.4\% on mathematics and 23.4\%--36.3\% on code, while maintaining or improving accuracy and achieving the best AUC across all code benchmarks. LoCo consistently improves the accuracy-efficiency trade-off on mathematics and transfers effectively to code generation despite math-only training, demonstrating the cross-domain generality of local preference learning for reasoning compression. The code and model will be publicly released at https://github.com/MuyaoYuan/LoCo.
Read the original paperMore in AI Reasoning
Browse all 51 papers →Scaling Lean Theorem Proving for Real-World Code Verification
Debargha Ganguly, Divya Raghunathan, Dejan Jovanović, Vikash Singh, Soonho Kong, Aaron Tomb, Aman Goel, Ferhat Erata, Sam Bayless
EasyProver tackles Lean proofs at scale and introduces harder verification challenges grounded in real Rust code.
Self-Improvement with Auxiliary Rewards: Learning from Verified Traces and Evaluating Transfer
Minju Gwak, Alan Ritter
The study finds that how a model reasons can help guide its learning, even when multiple reasoning traces reach the same correct answer.
Amortizing Sequential Experimental Design in Reasoning LLMs
Anita Kriz, Antonin Berthon, Nicolás Astorga, Mihaela van der Schaar
The paper teaches LLMs to ask more informative questions, making sequential information gathering cheaper and more transferable.