World of SWECraft: Agent Collaboration at Scale via Simulated Software Worlds
AuthorsShannon Zejiang Shen, Yapei Chang, Qiyao Liang, Benjamin Newman, Jenny Liang, Rulin Shao, Ziqiao Ma, Junjie Oscar Yin, Yung-Sung Chuang, Weijia Shi, Yejin Choi, Luke Zettlemoyer, Pang Wei Koh, Mohit Iyyer, Tongshuang Wu, David Sontag
AffiliationsOverview of Software World. Left: we build a software world from real Python libraries that depend on one another. Agents maintain these libraries and work together to find tasks and make the code fas · University of Maryland, College Park · University of Washington · Carnegie Mellon University · University of Michigan
The paper builds simulated software ecosystems to test how AI agents collaborate and scale their work.
Key results
The simulations covered over 10K agent-hours.
Total sessions across the reported experiments.
Total tokens used across the simulations.
Improvement achieved by the strongest Gemini 3.8 Flash run.
Session traces released with the framework.
What the paper found
Software World is a framework for studying AI collaboration through the ongoing maintenance of real, interdependent Python packages. Agents choose their own efficiency improvements, work through code repositories and issue discussions, and can coordinate across package boundaries; SW-Eval then measures whether their changes speed up hidden downstream packages while preserving correctness, using pytest-codspeed and Callgrind instruction counts. Across three software worlds and models from Google Gemini, Anthropic Claude, and OpenAI GPT, the experiments cover over 10K agent-hours, 55K sessions, and 180B tokens. The strongest result, from Gemini 3.8 Flash, improves downstream efficiency by up to 31%. The findings show that collaboration can uncover work agents miss alone, but simply prompting more communication does not guarantee better performance. Clear package ownership reduces duplicated effort, while unassigned teams can repeat changes; shared budgets can also go unused or fund work that does not improve the held-out benchmark. Some strong models stop searching despite remaining opportunities, highlighting task discovery as a distinct challenge for autonomous coding agents. The release includes 40K agent-session traces for further study.
Original abstract
As teams of AI agents take on larger projects and long-standing open problems, agent collaboration offers another dimension for scaling inference-time compute. To understand what makes such collaboration effective and scalable, we develop Software World, a framework for simulating multi-agent coworking in realistic software ecosystems. It comes with a pipeline to create environments of real, interdependent Python packages of different topics, sizes, and dependency structures, which we call software worlds. In a simulation, agents maintain packages in the world with a high-level goal of improving software efficiency, and they are free to discover tasks to work on and collaborate with others. We evaluate agents' work in a world collectively by measuring speedup in hidden dependent packages. The framework allows us to study different elements of collaboration by varying communication, ownership, and resource allocation. We run simulations in three different worlds, totaling more than 10,000 hours of agent work across eight models. Overall, performance improvement scales with inference-time compute, with the best model improving downstream efficiency by up to 31%. We show that most agents can find meaningful tasks via collaboration, but some frontier models can prematurely stop looking for improvements and make little progress. We release the simulation framework and traces to support future studies of agent collaboration at https://szj.io/sw.
Read the original paperMore in AI Agents
Browse all 65 papers →AutoEnvScaling: Automating the Data Flywheel with Terminal Agents
Simon Yu, Baolin Peng, Bo Liu, Zhengyan Shi, Zhaoyang Wang, Hao Cheng, Qianhui Wu, Wenlin Yao, Natasha Jaques, Weiyan Shi, Jianfeng Gao
AutoEnvScaling has agents create the training environments that help terminal agents get better at solving tasks.
DuoDiscover: Scaling Autonomous Discovery Beyond the Evaluation Bottleneck
Shubham Agarwal, Akshat Gupta, Mike Heddes, Dilara Soylu, Rui Meng, Patrick Tser Jern Kon, Jiefeng Chen, Mert Cemri, Shu Liu, Abe Gonzalez, Akanksha Jain, Bhavana Dalvi Mishra, Linfeng Song, Vilobh Meshram, Xiangru Tang, Jinsung Yoon, Aditya Parameswaran, Ion Stoica, Vahab Mirrokni, Burak Gokturk, Parthasarathy Ranganathan, Tomas Pfister, Nanyun Peng, Chun-Liang Li
DuoDiscover uses a fleet of AI agents and fast, continuously checked proxy tests to find engineering improvements when full evaluations are too slow.
CUAWright: A Minimal Unified Interface for Digital Agents
Yadong Lu, Theodore Lee, Yifei Li, Lawrence Keunho Jang, Tianci Xue, Yu Su, Huan Sun, Ahmed Hassan Awadallah
CUAWright argues that giving computer-use agents a simple programmable terminal instead of GUI-specific tools makes them substantially more capable, efficient, and adaptable.