Accelerate Bench: Evaluating Agents for Application-Level Program Optimization
AuthorsBowen Liu, Yuhan Li, Nuo Chen, Weile Luo, Miao Peng, Qintong Zhang, Qifan Zhang, Jia Li, Ziqi Gao, Lydell Huang
AffiliationsTencent Hy Team · The Hong Kong University of Science and Technology (Guangzhou) · Peking University · The Hong Kong University of Science and Technology · Tsinghua University
A new benchmark tests whether coding agents can make whole scientific programs faster without breaking them.
Key results
Workload-defined tasks in Accelerate Bench.
Coding-agent and harness combinations evaluated on benchmark version 1.1.
Highest reported Score@1 among the evaluated agents.
The two leading agents each reach this threshold on 22 or more tasks.
Approximate runtime improvement from deferred loading when broad simulator dispatch is enabled.
What the paper found
Accelerate Bench evaluates application-level optimization rather than isolated code edits: agents receive 32 CPU workloads with fixed correctness requirements and may change algorithms, source code, build settings, and runtime configuration, then face private verification. Across 16 agents, Score@1—which rewards correctness-gated fractions of baseline runtime saved—ranges from 25.5% to 67.4%; Anthropic’s Claude Sonnet 5.5 leads, ahead of agents including OpenAI’s GPT-6.1 Sol and DeepSeek-V4 models. The two leading agents achieve at least 2× speedup on 22 or more tasks, while most other agents do so on 8 to 17. Trajectory analysis and controlled replays show why application-wide coordination matters: identical GROMACS source runs at 0.29× to 2.12× speedup under different build and runtime settings. In QiboTN, deferred tensor-library loading improves runtime by only 1.01× alone, but by 6.4× when combined with broad dispatch to a faster simulator. The benchmark’s central contribution is to score complete verified workflows and expose interactions among code, configuration, and execution—not just whether an agent produces a locally plausible optimization.
Original abstract
Application optimization requires coding agents to discover useful changes and coordinate them across a complete program while preserving the required computation. We introduce Accelerate Bench, a benchmark of 32 CPU optimization tasks adapted from scientific applications and supercomputing competitions. Each task provides a working program and specifies the workload, correctness requirements, timing boundaries, and permitted edits, leaving agents to choose where and how to optimize. Public testing supports iterative development, and a separate verifier scores the final submission. On benchmark version 1.1, we evaluate 16 coding agents, each pairing a model with a harness. Their Score@1 ranges from 25.5% to 67.4%. Most of their programs are correct, and the size of the speedup separates them: the two leading agents are at least 2x faster on 22 or more of the 32 tasks, while the other fourteen reach 2x on 8 to 17 and leave 125 of their 412 correct outcomes below 1.25x. Controlled replays show why coordination across an application matters: identical source code runs between 0.29x and 2.12x depending only on build and runtime settings, and one code change is worth 1.01x or 6.4x depending on another. Accelerate Bench pairs executable workloads with agents' programs and trajectories to evaluate both their gains and the process behind them.
Read the original paperMore in AI Benchmarks
Browse all 60 papers →DrawScore: Evaluating Generated CAD Models against Engineering Drawings
Haorui Chen, Shiteng Lu, Zhanyu Guo, Zaixi Li, Yunhe Ni, Yaosong Du, Yichen Xiang, Xiande Huang, Rui Huang
DrawScore checks generated CAD models through engineering-style views, making small but manufacturing-critical mistakes harder to overlook.
PReMind: When Personal Memory Meets Visual Hazards
Yeongtak Oh, Donghyeok Lee, Joonhee Oh, Jaewon Jeong, Sungroh Yoon
PReMind tests and trains visual assistants to connect what they see with the personal context needed to give safer advice.
AutoVend: Benchmarking LLM Agents on Long-Horizon Autonomous Vending Operations
Qiming Shi, Yikun Li, Beidi Luan, Rui Sun, Sinuo Wang, Delin Wang, Yan Ma, Jing Li, Daxin Jiang, Zuo Bai
AutoVend tests whether LLM agents can run vending-machine businesses profitably over six months—and finds that most struggle to beat a simple inventory baseline.