FrontierCS 2: A Benchmark for Computer Science R&D
AuthorsQiuyang Mang, Wenhao Chai, Runyuan He, Dongruixuan Li, Zihan Zheng, Yujun He, Xuanyi Zhou, Hao Wang, Siddhant Ray, Bo Peng, Ruyi Ji, Shuo Yang, Haocheng Xi, Boru Chen, Xinyang David Han, Melissa Pan, Mayank Mishra, Boxi Yu, Sijia Liu, Kerem Akillioglu, Yi Pan, Zirui Wang, Mert Cemri, Diogo Antunes, Yichuan Wang, Aiwei Ivy Zhang, Zhenyang Chen, Qizheng Zhang, Kaiyuan Liu, Wenyuan Huang, Sherry Ruan, Shu Liu, Dacheng Li, Zerui Cheng, Zeyu Shen, Shang Zhou, Hanfei Chen, Yuzhong Zhang, Bohan Lyu, Qu Ao, Han Zheng, Chonghe Jiang, Jingzhuo Hu, Xiaokun Chen, Kunjun Li, Zhiqing Zhong, Haoye Tian, Tong Zhao, Yunzhong He, Hanchen Li, Ofir Press, Jiantao Jiao, Kunle Olukotun, Karthik R. Narasimhan, Jingbo Shang, Koushik Sen, Matei Zaharia, Alex Dimakis, Ion Stoica, Joseph E. Gonzalez, Alvin Cheung
AffiliationsUC Berk
FrontierCS 2 tests whether AI agents can improve real research codebases across a wide range of computer science tasks.
Key results
Open-ended repository-improvement tasks in the benchmark.
Computer-science domains represented.
Agents evaluated, each using a model and harness.
Share of its runs that beat the reference on every metric and held-out workload.
No evaluated agent passed these tasks.
Muse Wonder 1.0 result on the 67-million-entry database.
What the paper found
FrontierCS 2 tests whether AI agents can improve real computer-science research artifacts. Each task starts from a published paper’s repository with its key contribution removed; agents submit iterative patches, receive development-suite feedback, and are judged privately against the original implementation on held-out workloads. A pass requires beating the reference on every metric and workload. The benchmark covers 48 tasks across 13 domains and evaluates 12 agents, including Claude Opus 5.5, GPT-6 Astra, Gemini 4 Argon, and DeepSeek V4.1 Flash. Claude Opus 5.5 leads with a 58% pass rate, while no agent passes 16 tasks. The researchers also introduce the FrontierCS Capabilities Index, which combines metric outcomes and score margins to place agents and task difficulty on a shared scale; the median task is about as difficult as the strongest agent can handle. A case study illustrates both the potential and the limits of these evaluations: Muse Wonder 1.0 uses two checksums to reduce QuarterPIR replies to 1280 bytes per lookup on a 67-million-entry database, while the benchmark’s all-metrics pass rule can reject improvements that still miss a quality threshold.
Original abstract
Can LLM agents make contributions to frontier computer science R&D? We introduce FrontierCS 2, a benchmark of 48 open-ended tasks. We take a paper and the code its authors released and delete the paper's contribution. The deleted code becomes the human reference that sets the bar. Earlier open-ended benchmarks give each agent a problem statement and grade one program after the run. FrontierCS 2 places the agent inside a research environment. The agent inherits a working repository and may submit a patch at any point. For each patch, a judge returns a development-suite score but keeps the evaluator, the held-out final suite and the reference private, and grades the final submission on that suite against the reference. The tasks span 13 domains, from machine-learning systems and high-performance computing to databases, security and robotics. Beyond these 48 tasks, FrontierCS 2 keeps a private set of more than 100 tasks, to be announced in the future. We evaluate 12 agents. The best one is Claude Opus 5.5, passing 58% of its runs. No run passes on 16 tasks. A higher score is not the same as a pass: most runs that beat the reference on the combined final score still fall short on some metric or workload. On a shared capability scale for agents and tasks, the median task is roughly as difficult as what the strongest agent can handle.
Read the original paperMore in AI Benchmarks
Browse all 51 papers →HandsOn: A Large-Scale Simulation Benchmark for Generalizable Dexterous Manipulation
Tianwei Luo, Yiming He, Hongjin Wu, Ting Pan, Debin Ren, Haorui Guo, Junlin Yang, Yuxin Zuo, Zhehao Kou, Kaisen Yang, Zhuoqun Lai, Yi Ding, Jiale Yu, Yuchen Zhang, Zijie Qiu, Haozhan Li, Ning Ding
HandsOn is a large benchmark for testing whether dexterous robots can generalize manipulation skills across hands, objects, and tasks.
Last Secure Code Benchmark: Is the Code You Vibe Secure?
Tianyi Wu, Chao Ge, Peng Chen, Fengji Zhang, Yongxing Wu, Yuchong Xie, Jian Li, Chaoyuan Peng, Kaikai Zhang, Dong Huang, Mingzhe Du, Yukai Huang, Jiaheng Zhang, Terry Yue Zhuo, See-Kiong Ng
A new benchmark finds that coding agents often produce functional code that still fails to protect against real vulnerabilities.
LifeSearch: Benchmarking Deep Research Agents for Real-World Life and Productivity
Yibo Wang, Senjie Jin, Yansong Ning, Xinbing Liang, Haotian Luo, Jiazheng Zhang, Yuxuan He, Will Wu, Clarenceai
LifeSearch tests how well deep-research agents handle everyday and professional questions—and finds that even top systems still have substantial room to improve.