NTH

Self-Improvement with Auxiliary Rewards: Learning from Verified Traces and Evaluating Transfer

AuthorsMinju Gwak, Alan Ritter

AffiliationsGeorgia Institute of Technology · mgwak9@gatech.edu, alan.ritter@cc.gatech.edu

November 8, 2026 2 min read
Watch on YouTube
The one-line take

The study finds that how a model reasons can help guide its learning, even when multiple reasoning traces reach the same correct answer.

Key results

30
AIME2026 adaptation problems

Qwen3-1.7B online adaptation used majority agreement rather than reference answers.

2.08
UID-Conj TTRL gain over TTRL

Percentage-point gain in four-benchmark macro-average accuracy.

1.93
UID-Conj weighted-SFT gain over uniform SFT

Percentage-point gain on the same verified-response pool.

1.24
CriticBench verdict accuracy gain

Percentage-point gain for UID-Conj over TTRL.

1.58
CriticBench incorrect-class F1 gain

Percentage-point gain for UID-Conj over TTRL.

What the paper found

This paper asks whether models can learn more from reasoning traces that reach the same correct answer. It compares Self-Certainty, or SC, which scores predictive confidence, with UID-Conj, which rewards traces combining global entropy variation and local smoothness. In online test-time reinforcement learning, the auxiliary score adjusts rewards only among responses agreeing with the majority answer; offline, it guides preferences between verified correct traces in DPO or assigns weights in supervised fine-tuning. On Qwen3-1.7B, adapting with no reference answers on 30 AIME2026 problems, UID-Conj improves the four-benchmark macro-average by 2.08 percentage points over unmodified TTRL. Its clearest result is offline weighted SFT: it beats uniform weighting on the same response pool by 1.93 percentage points. The gains also transfer beyond mathematics: versus TTRL, UID-Conj improves CriticBench verdict accuracy by 1.24 percentage points and incorrect-class F1 by 1.58 percentage points, while also improving LabBench2 accuracy. The signals are not interchangeable: SC more often selects traces rated highly for reasoning quality, while UID-Conj yields stronger training results in several settings. Trace quality was assessed separately by OpenAI’s GPT-6 Luna, underscoring that the paper evaluates both which reasoning traces appear better and which ones actually help a model learn.

Original abstract

Answer-level rewards cannot distinguish reasoning traces with equally correct outcomes. We study whether auxiliary trajectory preferences improve small-language-model self-improvement and transfer under limited supervision. We compare confidence-based (SC) and entropy-based metrics (UID-Conj.), which combines global entropy variation and local smoothness, through online reward augmentation and offline training on correct responses. Online experiments adapt Qwen3-1.7B on 30 AIME problems without reference answers; offline experiments use historical AIME problems. UID-Conj. improves the online macro-average across the adaptation benchmark and three transfer benchmarks by 2.08 percentage points over unmodified TTRL. The strongest evidence comes from offline weighted SFT: UID-Conj. exceeds uniform weighting on the same response pool by 1.93 points, with a paired prompt-bootstrap 95\% CI of [0.44, 3.44]. Benefits are task-dependent: SC rewards judged trace quality and instruction following, while UID-Conj. rewards yield stronger training outcomes in several settings. We find that auxiliary preferences can expose learning value among equally correct traces not captured during inference time, motivating their evaluation through downstream gains.

Read the original paper

More in AI Reasoning

Browse all 51 papers →
01Reasoning

Scaling Lean Theorem Proving for Real-World Code Verification

Debargha Ganguly, Divya Raghunathan, Dejan Jovanović, Vikash Singh, Soonho Kong, Aaron Tomb, Aman Goel, Ferhat Erata, Sam Bayless

EasyProver tackles Lean proofs at scale and introduces harder verification challenges grounded in real Rust code.

Read analysis