Can Generalist Agents Automate Data Curation?

This paper asks whether generalist coding agents, including Anthropic’s Claude Code, OpenAI’s Codex, and open-source backbones run through OpenHands, can automate training-data curation as an iterative research loop rather than a one-shot selection problem. The authors introduce Curation-Bench, a terminal-based benchmark that fixes the model, training recipe, evaluation suite, and contamination checks while giving agents control only over executable data policies. On a 10k-example selection task from LLaVA-665K for fine-tuning LLaVA-1.5-7B, open-prompt agents reliably execute the loop and outperform random selection and published baselines such as ICONS and ARDS, with Claude Code reaching 33.7 average score versus 32.5 for the best random run and recovering 59% of the full-data gain using only 1.5% of the pool. The key failure is that open-ended prompting produces local heuristic edits—source ratios, length thresholds, and seed sweeps—rather than new policy families. Light scaffolds broaden the search vocabulary but do not raise the best score, while heavy scaffolds that force evidence-grounded or method-grounded iteration do: the strongest scaffold composes an EL2N-style top-loss selector with a p95 assistant-loss noise filter and reaches 34.9, exceeding the open-prompt maximum and even a 100k ARDS baseline. Longer sessions also keep improving up to 50 iterations, and the benchmark extends to DataComp Small, where the agent beats the top-30% CLIP ViT-L/14 filtering baseline, showing the framework generalizes beyond instruction tuning.