$τ^τ$-Bench: An Environment for End-To-End, Realistic Agent Construction

ττ-bench evaluates whether coding agents can construct deployable customer-service agents rather than merely operate prebuilt ones. Each task supplies scattered multimodal business evidence, an interactive client holding undocumented requirements, a potentially defective production REST API, inherited code, and fixed model and serving-cost constraints; the constructed agent is then deployed against held-out simulated customers. Across 53 tasks in airline, retail, telecom, and banking, Claude Opus 5 under Anthropic’s Claude Code achieved the strongest result at 23.9%, versus an expert-authored reference ceiling of 82.2%. The benchmark contains 2,868 evidence artifacts and over 5.5M tokens, forcing systems to reconcile documents, transcripts, spreadsheets, screenshots, recordings, and conflicting records. Agents usually defaulted to a single LLM tool loop—92% of builds—while searching instead of reading, rarely interviewing the client, rewriting inherited systems without testing them, mishandling quiet API defects, and under-optimizing model budgets. On client-enabled tasks, builds asking four or more questions averaged 0.50, compared with 0.16 for builds asking none. A controlled telecom intervention shows that a one-line design hint—route by intent and review tool calls—raised performance from 31% to 67%, indicating that architecture selection and requirements elicitation, not only code generation, are major bottlenecks. The roster also included OpenAI’s GPT-5.6-sol and Google’s Gemini models, highlighting that current frontier coding systems can produce runnable agents but remain far from reliable end-to-end engineering.