Self-Improvement with Auxiliary Rewards: Learning from Verified Traces and Evaluating Transfer
AuthorsMinju Gwak, Alan Ritter
AffiliationsGeorgia Institute of Technology · mgwak9@gatech.edu, alan.ritter@cc.gatech.edu
The study finds that how a model reasons can help guide its learning, even when multiple reasoning traces reach the same correct answer.
Key results
Qwen3-1.7B online adaptation used majority agreement rather than reference answers.
Percentage-point gain in four-benchmark macro-average accuracy.
Percentage-point gain on the same verified-response pool.
Percentage-point gain for UID-Conj over TTRL.
Percentage-point gain for UID-Conj over TTRL.
What the paper found
This paper asks whether models can learn more from reasoning traces that reach the same correct answer. It compares Self-Certainty, or SC, which scores predictive confidence, with UID-Conj, which rewards traces combining global entropy variation and local smoothness. In online test-time reinforcement learning, the auxiliary score adjusts rewards only among responses agreeing with the majority answer; offline, it guides preferences between verified correct traces in DPO or assigns weights in supervised fine-tuning. On Qwen3-1.7B, adapting with no reference answers on 30 AIME2026 problems, UID-Conj improves the four-benchmark macro-average by 2.08 percentage points over unmodified TTRL. Its clearest result is offline weighted SFT: it beats uniform weighting on the same response pool by 1.93 percentage points. The gains also transfer beyond mathematics: versus TTRL, UID-Conj improves CriticBench verdict accuracy by 1.24 percentage points and incorrect-class F1 by 1.58 percentage points, while also improving LabBench2 accuracy. The signals are not interchangeable: SC more often selects traces rated highly for reasoning quality, while UID-Conj yields stronger training results in several settings. Trace quality was assessed separately by OpenAI’s GPT-6 Luna, underscoring that the paper evaluates both which reasoning traces appear better and which ones actually help a model learn.
Original abstract
Answer-level rewards cannot distinguish reasoning traces with equally correct outcomes. We study whether auxiliary trajectory preferences improve small-language-model self-improvement and transfer under limited supervision. We compare confidence-based (SC) and entropy-based metrics (UID-Conj.), which combines global entropy variation and local smoothness, through online reward augmentation and offline training on correct responses. Online experiments adapt Qwen3-1.7B on 30 AIME problems without reference answers; offline experiments use historical AIME problems. UID-Conj. improves the online macro-average across the adaptation benchmark and three transfer benchmarks by 2.08 percentage points over unmodified TTRL. The strongest evidence comes from offline weighted SFT: UID-Conj. exceeds uniform weighting on the same response pool by 1.93 points, with a paired prompt-bootstrap 95\% CI of [0.44, 3.44]. Benefits are task-dependent: SC rewards judged trace quality and instruction following, while UID-Conj. rewards yield stronger training outcomes in several settings. We find that auxiliary preferences can expose learning value among equally correct traces not captured during inference time, motivating their evaluation through downstream gains.
Read the original paperMore in AI Reasoning
Browse all 51 papers →Scaling Lean Theorem Proving for Real-World Code Verification
Debargha Ganguly, Divya Raghunathan, Dejan Jovanović, Vikash Singh, Soonho Kong, Aaron Tomb, Aman Goel, Ferhat Erata, Sam Bayless
EasyProver tackles Lean proofs at scale and introduces harder verification challenges grounded in real Rust code.
Think Less, Locally: Local Preference Learning for Reasoning Compression
Muyao Yuan, Liyan Xu, Weizhan Zhang, Muyan Jiao, Jiangnan Li, Jinchao Zhang, Jie Zhou
LoCo teaches reasoning models when to stop thinking, cutting response length while preserving or improving accuracy.
Amortizing Sequential Experimental Design in Reasoning LLMs
Anita Kriz, Antonin Berthon, Nicolás Astorga, Mihaela van der Schaar
The paper teaches LLMs to ask more informative questions, making sequential information gathering cheaper and more transferable.