NTH
AI Reasoning

Inducing Process Supervision from Outcome-Only Reinforcement Learning

AuthorsShengda Fan, Xin Cong, Zhong Zhang, Haotian Chen, Yankai Lin

AffiliationsRenmin University of China · Tsinghua University · University of Electronic Science and Technology of China · Shanghai Jiao Tong University

October 20, 2026 2 min read
Watch on YouTube

Resources

View on Hugging Face Read the original paper on arXiv GitHub repository
The one-line take

TIPS trains models to judge their own reasoning steps using only outcome feedback, avoiding the need for costly step-by-step annotations.

Key results

3.2K
Outcome-labeled training trajectories

Used to train TIPS-Qwen3-4B-Thinking for ProcessBench.

85.2
ProcessBench F1

Score for TIPS-Qwen3-4B-Thinking.

49.1
AgentProcessBench First-Error Accuracy

Average score for TIPS-Qwen3-4B-Thinking.

73.3
Best-of-8 average accuracy

TIPS-Qwen3-4B-Thinking reranking result.

What the paper found

TIPS, or Thinking-Induced Process Supervision, trains a generative process reward model using only trajectory-level outcome labels. The model generates a chain of thought, labels each reasoning step, and predicts whether the final answer is correct; GRPO rewards only correct outcome predictions, applying the resulting group-relative advantage across the entire response. This lets outcome feedback reinforce useful step-checking behavior without human step annotations or Monte Carlo process labels. On ProcessBench, TIPS-Qwen3-4B-Thinking reaches 85.2 F1 from just 3.2K outcome-labeled trajectories, exceeding evaluated trained PRMs and prompt-only judges including OpenAI’s GPT-5.4-Instruct and Anthropic’s Claude-4.7-Opus, though it trails OpenAI’s o1-mini. On AgentProcessBench, the Qwen3-4B-Thinking model’s average First-Error Accuracy rises from 43.7 to 49.1. In Best-of-8 reranking, TIPS reaches 73.3 average accuracy. Tests across Qwen, DeepSeek, Llama, and SmolLM backbones suggest the approach generalizes, but its process gains depend on the model’s ability to reason in its chain of thought and on local step correctness being closely tied to final success.

Original abstract

Process reward models (PRMs) have become a key component for LLMs, as their step-level feedback supports both post-training and test-time reasoning. However, training strong PRMs remains costly: human step annotation is difficult to scale, while Monte Carlo estimation is computationally expensive and can drift from the intrinsic correctness of steps. To get effective PRMs at low cost, we introduce TIPS (Thinking-Induced Process Supervision), an outcome-only reinforcement learning (RL) framework for training generative PRMs. In TIPS, the model generates a chain-of-thought (CoT) followed by step-level labels and an outcome label. The reward depends solely on whether the predicted outcome matches the ground truth, and the resulting group-relative advantage is used to optimize the entire generated response. Intuitively, when checking intermediate steps helps determine the outcome, more accurate checks can lead to better outcome judgments and higher rewards. Outcome-only RL can therefore reinforce step-level verification without explicit process supervision. We validate the effectiveness of TIPS across math and agent benchmarks and four backbone families. Notably, TIPS-Qwen3-4B-Thinking-2507 reaches 85.2 F1 on ProcessBench with only 3.2K outcome-labeled trajectories, surpassing all evaluated trained PRMs and strong prompt-only judges such as GPT-5.4-Instruct and Claude-4.7-Opus, while still trailing o1-mini. Code and data are available at https://github.com/RUCBM/TIPS.

Read the original paper

More in AI Reasoning

Browse all 45 papers →