S3AC: Stabilized, Selective, and Scalable Actor-Critic for LLM Reasoning
AuthorsShuo Shen, Guangran Cheng, Chengqi Lyu, Xiangsen Wang, Yanan Sun, Wenwei Zhang
AffiliationsShanghai AI Laboratory · Independent Researcher
S3AC trains a critic to improve token-level feedback for LLM reasoning and uses it to select more useful prompts, aiming to make reinforcement learning more effective and efficient.
Key results
Intern-S2-Preview-35B improves from 76.4 to 80.5 across seven benchmarks versus GRPO-X.
Intern-S2-397B improves from 85.1 to 87.3 across five benchmarks versus GRPO-X.
S3AC scores 80.9 versus 70.4 for the strongest tested critic-free baseline.
What the paper found
S3AC replaces response-level reward signals with a learned critic that estimates prompt difficulty before generation and assigns token-level advantages during reasoning. Its recipe pretrains the critic on trajectories from policies of different strengths, initializes the value head conservatively, and uses generalized advantage estimation with CISPO actor updates. Difficulty-Aware Prompt Sampling, or DAPS, combines critic estimates with past verifier outcomes to favor prompts of intermediate difficulty before spending compute on rollouts. Across Qwen2.5-Math-1.5B, DeepSeek-R1-Distill-1.5B, and Intern-S2 models, S3AC improves group-rollout results over GRPO-X: on Intern-S2-Preview-35B, the seven-benchmark average rises from 76.4 to 80.5, a 4.1-point gain; on Intern-S2-397B, it rises from 85.1 to 87.3, a 2.2-point gain. With single rollouts on the 35B model, S3AC scores 80.9 versus 70.4 for the strongest tested critic-free baseline, a 10.5-point advantage, and sustains training while those baselines decline. DAPS further improves results over uniform prompt sampling and reduces generated responses later discarded by group filtering, though maintaining a critic adds training and inference cost.
Original abstract
Critic-free methods are widely used in reinforcement learning with verifiable rewards (RLVR) for long chain-of-thought (CoT) reasoning, avoiding difficult and costly critic training. Common alternatives lack token-level credit assignment and typically rely on grouped rollouts to estimate prompt-specific baselines. A critic can address both limitations, but long-CoT critic training is hindered by insufficient coverage of trajectories with diverse lengths and early gradients dominated by the newly initialized value head. We introduce S3AC (Stabilized, Selective, and Scalable Actor-Critic) for LLM reasoning, built around a practical critic-training recipe. The trained critic provides token-specific advantages and prompt-specific baselines. Using the prompt-level baseline to estimate difficulty, we propose Difficulty-Aware Prompt Sampling (DAPS). DAPS combines this estimate with outcome history to select prompts before generation, reducing rollout waste without offline difficulty initialization. S3AC improves group-rollout performance across dense and mixture-of-experts models from 1.5B to 397B parameters, outperforming a modern GRPO variant (GRPO-X). At 35B, it sustains single-rollout training while tested critic-free baselines deteriorate after initial gains.
Read the original paperMore in Large Language Models
Browse all 93 papers →Advancement or Acceleration? How Data Curation Improves LLM Training
Jiaxing Wang, Zicheng Zhang, Deping Xiang, Jin Xu, Yong Li, Chao Liu, Pengzhang Liu, Qixia Jiang
This paper asks whether better-curated data helps LLMs learn more or simply helps them learn the same things faster.
Watermark Basins: Black-Box Detection of Distilled LLMs
Hongyan Chang, Hady Elsahar, Tom Sander, Pierre Fernandez, Saeed Mahloujifar, Sylvestre-Alvise Rebuffi, Tomáš Souček, Tuan Tran, Valeriu Lacatusu, Surya Parimi, Rashel Moritz, Vanessa Stark, Alexandre Mourachko
A new black-box method detects when an LLM may have been distilled from watermarked outputs by looking for a persistent signal in how its responses relate to model gradients.
Belief-Trajectory Energy: Measuring the Path to a Prediction
Jiahao Ying, Wei Tang, Boxian Ai, Yaoning Wang, Haotian Chen, Wenhe Sun, Caijun Xu, Haozhan Cai, Changyi Xiao, Yixin Cao
Belief-Trajectory Energy tracks how an LLM’s predictions change across layers to reveal information that its final answer alone can miss.