Autodata: An agentic data scientist to create high quality synthetic data

Autodata, from FAIR at Meta, is a general framework that turns an AI agent into a data scientist that iteratively creates, evaluates, and revises synthetic training data, then meta-optimizes its own prompting strategy. In the paper’s Agentic Self-Instruct implementation, a Kimi-K2.6 orchestrator coordinates a challenger, weak solver, strong solver, and judge to generate examples that separate capability levels, using Qwen3.5-4B as the weak model and Qwen3.5-397B-A17B as the strong model. On computer science papers from S2ORC, the agent needed 6.59 rounds on average to accept an example, and the accepted set shifted weak-solver quality from 0.677 to 0.458 while raising strong-solver quality from 0.696 to 0.772; training Qwen3.5-4B with GRPO on 1.3k Agentic examples lifted held-out mean@3 from 0.630 to 0.774 on the CoT test and from 0.366 to 0.632 on the harder Agentic test. On legal reasoning from Pile of Law, the same loop reshaped weak rollouts from degenerate all-zero outputs into a usable learning signal, and GRPO training on 2.8k Agentic examples improved PRBench-Legal from 0.343 to 0.393 under Kimi-K2.6 grading, surpassing the larger 397B baseline at 0.358. In scientific reasoning over mathematical objects, Agentic data achieved the best overall avg@8 on the combined validation set at 71.86%, and reduced truncation rates at a 65,536-token budget from 23.75% to 4.09%; a separate meta-optimization loop further raised the CS validation pass rate from 62.1% to 79.6% over 233 iterations.