ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks

ProgramDistill, developed with Microsoft, introduces a benchmark for reference-guided software engineering: an agent observes a working interactive web app, infers its behavior without source access, repairs an incomplete version, and validates the result through replayable browser traces. Its automated mine-craft-patch pipeline, built with GPT-5.6 Sol, factorizes 26 applications into 1975 replay-verified behaviors and 4063 tasks, using prerequisite lineages, source masks, deterministic execution, and gold patches as executable verifiers. The benchmark spans atomic repair, cumulative restoration, and full-application reconstruction, with difficulty controlled by restoration depth. On partial reconstruction, OpenAI’s GPT-6 Astra achieves an 84.3% mean binary score, ahead of Anthropic’s Claude Opus 5 at 68.7%, but Astra’s performance falls from 100% at depth 1 to 64.0% at depth 8. In full-application reconstruction, Astra reaches 49.2% cumulative workflow recovery, compared with 28.8% for Claude Opus 5. The study also evaluates GPT-5.3 Codex, Google DeepMind’s Gemini models, and xAI’s Grok 4.6, showing that deeper tasks expose compositional failures: agents observe and validate less per required behavior as repair burden grows. ProgramDistill therefore serves both as a diagnostic benchmark and as a potential curriculum and reinforcement-learning environment for coding agents.