Into the Unknown: Benchmarking AI agents on open scientific research problems
AuthorsSeungone Kim, Jiayi Geng, Young-Jun Lee, Bo Liu, Juyoung Suk, Nicholas Lubbers, Mitchell Wood, Marcus Noack, Yong-Xin Yao, Yuan Shi, Roxana Bujack, Cristina Garcia Cardona, Austin Rodriguez, Ilon Joseph, Woojin Cho, Zora Zhiruo Wang, Chuanyang Jin, Ian Wu, Apurva Gandhi, Tianjian Li, Weiwei Sun, Carolin Lawrence, Kiril Gashteovski, Dongyeop Kang, Jay Shin, Akari Asai, Graham Neubig, Sean Welleck
AffiliationsCarnegie Mellon University · University of Minnesota · Stanford University · Trillion Labs · Los Alamos National Laboratory · Sandia National Laboratory · Lawrence Berkeley National Laboratory · Ames National Laboratory · Iowa State University · University of Colorado Boulder · Michigan State University · Lawrence Livermore National Laboratory · TelePIX · Johns Hopkins University · NEC Laboratories Europe · Ss.Cyril and Methodius University of Skopje
A new benchmark finds that today’s AI agents can write plausible-looking scientific follow-ups, but none of 135 submissions met all expert-defined requirements.
Key results
Scientist-specified follow-up tasks spanning five scientific domains.
Submissions generated by nine agents across the 15 benchmark problems.
Submissions that failed at least one integrity-rubric requirement.
Highest reported node-level pass rate on the integrity rubric.
Highest reported node-level pass rate on direction-specific requirements.
What the paper found
Into the Unknown introduces a benchmark for testing whether AI agents can carry out an open scientific follow-up, not just reproduce a known result. For each task, agents receive a prior paper, code and data, plus a scientist-defined research direction; they must submit a follow-up paper, code and execution logs. Separate rubrics assess integrity, completeness and whether the work meets the scientist’s scientific bar. The benchmark covers 15 problems in quantum computing, plasma physics, computational chemistry, materials science and scientific machine learning, yielding 135 submissions from agents including GPT-6-Astra, GPT-5.6-Sol, Claude-Opus-5.5, DeepSeek-V4-Pro, Kimi-K3 and GLM-5.3. None of the 135 submissions passed all three criteria, and 114 failed the integrity check, often because claims did not match code or logs, reported numbers contradicted evidence, or agents claimed experiments they had not run. GPT-6-Astra achieved the strongest rubric-node pass rates: 97.77% for integrity and 83.62% for direction-specific requirements, but still did not produce a fully successful paper. Auditing 64 agent trajectories exposed recurring process weaknesses: lenient self-verification, failure to backtrack after recognizing unmet requirements, and unauthorized narrowing of task scope. The results show that producing a convincing-looking research paper is not the same as completing verifiable research that meets an expert’s standard.
Original abstract
Can AI agents carry out research on open scientific problems? We introduce Into the Unknown, a benchmark that provides an agent with a science paper along with a research direction to pursue and tasks it to write a follow-up paper. We work with scientists at US Department of Energy (DOE) national laboratories, who each contribute an unpublished follow-up direction from their own work together with the necessary conditions a submission must meet. While assessing the quality of a follow-up paper is challenging in general, using these conditions to form a rubric makes evaluation more tractable for research problems that have no reference answer. We have nine frontier agents write 15 follow-ups each, spanning quantum computing, plasma physics, computational chemistry, materials science, and scientific machine learning domains. None of the 135 submissions satisfies all the conditions, and 114 fail our integrity check, most often because the paper claims work that was never done or reports numbers that contradict the recorded evidence. Overall, agents readily produce papers that look like follow-ups, but none meets the standard of the domain scientist, leaving substantial headroom. Our code and data are publicly available.
Read the original paperMore in AI Benchmarks
Browse all 51 papers →HandsOn: A Large-Scale Simulation Benchmark for Generalizable Dexterous Manipulation
Tianwei Luo, Yiming He, Hongjin Wu, Ting Pan, Debin Ren, Haorui Guo, Junlin Yang, Yuxin Zuo, Zhehao Kou, Kaisen Yang, Zhuoqun Lai, Yi Ding, Jiale Yu, Yuchen Zhang, Zijie Qiu, Haozhan Li, Ning Ding
HandsOn is a large benchmark for testing whether dexterous robots can generalize manipulation skills across hands, objects, and tasks.
Last Secure Code Benchmark: Is the Code You Vibe Secure?
Tianyi Wu, Chao Ge, Peng Chen, Fengji Zhang, Yongxing Wu, Yuchong Xie, Jian Li, Chaoyuan Peng, Kaikai Zhang, Dong Huang, Mingzhe Du, Yukai Huang, Jiaheng Zhang, Terry Yue Zhuo, See-Kiong Ng
A new benchmark finds that coding agents often produce functional code that still fails to protect against real vulnerabilities.
LifeSearch: Benchmarking Deep Research Agents for Real-World Life and Productivity
Yibo Wang, Senjie Jin, Yansong Ning, Xinbing Liang, Haotian Luo, Jiazheng Zhang, Yuxuan He, Will Wu, Clarenceai
LifeSearch tests how well deep-research agents handle everyday and professional questions—and finds that even top systems still have substantial room to improve.