Advancement or Acceleration? How Data Curation Improves LLM Training
AuthorsJiaxing Wang, Zicheng Zhang, Deping Xiang, Jin Xu, Yong Li, Chao Liu, Pengzhang Liu, Qixia Jiang
AffiliationsZicheng Zhang · JD.com · University of Oxford
This paper asks whether better-curated data helps LLMs learn more or simply helps them learn the same things faster.
Key results
With this near-zero gap, mixture gains faded as training continued.
Gap when CommonCrawl was held out as the target.
TANDEM versus uniform allocation under the exogenous mixture gap.
Gap for the MetaMath–Mammoth target and OpenWebMath candidate pool.
Average across eight math and STEM benchmarks; full-pool continued pretraining scored 17.21.
Token count used for data selection with TinyLlama-1.1B.
What the paper found
This paper separates two effects of data curation: advancement, which improves a model’s best attainable performance, and acceleration, which reaches similar performance with less training. Its central finding is that advancement requires a correctable mismatch between the baseline training data and the evaluation target; when they align, sophisticated curation can speed learning but cannot improve the endpoint. The proposed ∆-test estimates this opportunity by comparing training and target losses from a single baseline run, measured at 50% of the training budget. In experiments with 160M-parameter GPT-style models trained on SlimPajama, a near-zero gap of ∆=0.07 corresponded to mixture methods’ early gains fading with more training; with an exogenous gap of ∆=1.26, TANDEM reduced CommonCrawl perplexity by 8.95 versus uniform sampling. For selection, the researchers used TinyLlama-1.1B and a 5B-token OpenWebMath pool: with a shifted MetaMath–Mammoth target and ∆=0.549, target-aware BLADE reached 31.31 average accuracy, compared with 17.21 for full-pool continued pretraining, while target-agnostic selectors did not show comparable gains. The results suggest checking for a train–target gap before investing in complex curation, and emphasize that choosing a useful target set helps determine which capabilities curation can improve.
Original abstract
While data curation is widely credited with improving large language model training, a basic question remains unresolved: does curation expand what a model can ultimately learn, or merely reach the same solution faster? In this paper, we formalize this distinction between advancement (improving the attainable performance) and acceleration (reaching the same performance with less data). Through an analytic characterization and real-world LLM experiments, we show that advancement requires a correctable train-target gap. When the baseline training and target objectives coincide, no curation policy improves the endpoint, although curation may still accelerate training. This reframes advancement as a property of the curation task rather than the curation method. Under a common gap-centric view of data mixture and selection, we operationalize this condition as the $Δ$-test, an observable discrepancy between the uncurated baseline's training and target losses that is computable prior to expensive curation. Across extensive experiments, an appreciably nonzero $Δ$ predicts persistent gains, whereas gains near $Δ=0$ tend to diminish as the training budget grows. Beyond individual methods, our findings recenter data curation on the task itself and point to target-set construction as a promising direction for shaping what curation can achieve.
Read the original paperMore in Large Language Models
Browse all 93 papers →S3AC: Stabilized, Selective, and Scalable Actor-Critic for LLM Reasoning
Shuo Shen, Guangran Cheng, Chengqi Lyu, Xiangsen Wang, Yanan Sun, Wenwei Zhang
S3AC trains a critic to improve token-level feedback for LLM reasoning and uses it to select more useful prompts, aiming to make reinforcement learning more effective and efficient.
Watermark Basins: Black-Box Detection of Distilled LLMs
Hongyan Chang, Hady Elsahar, Tom Sander, Pierre Fernandez, Saeed Mahloujifar, Sylvestre-Alvise Rebuffi, Tomáš Souček, Tuan Tran, Valeriu Lacatusu, Surya Parimi, Rashel Moritz, Vanessa Stark, Alexandre Mourachko
A new black-box method detects when an LLM may have been distilled from watermarked outputs by looking for a persistent signal in how its responses relate to model gradients.
Belief-Trajectory Energy: Measuring the Path to a Prediction
Jiahao Ying, Wei Tang, Boxian Ai, Yaoning Wang, Haotian Chen, Wenhe Sun, Caijun Xu, Haozhan Cai, Changyi Xiao, Yixin Cao
Belief-Trajectory Energy tracks how an LLM’s predictions change across layers to reveal information that its final answer alone can miss.