NTH

Accelerate Bench: Evaluating Agents for Application-Level Program Optimization

AuthorsBowen Liu, Yuhan Li, Nuo Chen, Weile Luo, Miao Peng, Qintong Zhang, Qifan Zhang, Jia Li, Ziqi Gao, Lydell Huang

AffiliationsTencent Hy Team · The Hong Kong University of Science and Technology (Guangzhou) · Peking University · The Hong Kong University of Science and Technology · Tsinghua University

November 21, 2026 2 min read
Watch on YouTube
The one-line take

A new benchmark tests whether coding agents can make whole scientific programs faster without breaking them.

Key results

32
CPU optimization tasks

Workload-defined tasks in Accelerate Bench.

16
Agents evaluated

Coding-agent and harness combinations evaluated on benchmark version 1.1.

67.4%
Claude Sonnet 5.5 Score@1

Highest reported Score@1 among the evaluated agents.

22
Tasks with at least 2× speedup

The two leading agents each reach this threshold on 22 or more tasks.

6.4
QiboTN conditional speedup

Approximate runtime improvement from deferred loading when broad simulator dispatch is enabled.

What the paper found

Accelerate Bench evaluates application-level optimization rather than isolated code edits: agents receive 32 CPU workloads with fixed correctness requirements and may change algorithms, source code, build settings, and runtime configuration, then face private verification. Across 16 agents, Score@1—which rewards correctness-gated fractions of baseline runtime saved—ranges from 25.5% to 67.4%; Anthropic’s Claude Sonnet 5.5 leads, ahead of agents including OpenAI’s GPT-6.1 Sol and DeepSeek-V4 models. The two leading agents achieve at least 2× speedup on 22 or more tasks, while most other agents do so on 8 to 17. Trajectory analysis and controlled replays show why application-wide coordination matters: identical GROMACS source runs at 0.29× to 2.12× speedup under different build and runtime settings. In QiboTN, deferred tensor-library loading improves runtime by only 1.01× alone, but by 6.4× when combined with broad dispatch to a faster simulator. The benchmark’s central contribution is to score complete verified workflows and expose interactions among code, configuration, and execution—not just whether an agent produces a locally plausible optimization.

Original abstract

Application optimization requires coding agents to discover useful changes and coordinate them across a complete program while preserving the required computation. We introduce Accelerate Bench, a benchmark of 32 CPU optimization tasks adapted from scientific applications and supercomputing competitions. Each task provides a working program and specifies the workload, correctness requirements, timing boundaries, and permitted edits, leaving agents to choose where and how to optimize. Public testing supports iterative development, and a separate verifier scores the final submission. On benchmark version 1.1, we evaluate 16 coding agents, each pairing a model with a harness. Their Score@1 ranges from 25.5% to 67.4%. Most of their programs are correct, and the size of the speedup separates them: the two leading agents are at least 2x faster on 22 or more of the 32 tasks, while the other fourteen reach 2x on 8 to 17 and leave 125 of their 412 correct outcomes below 1.25x. Controlled replays show why coordination across an application matters: identical source code runs between 0.29x and 2.12x depending only on build and runtime settings, and one code change is worth 1.01x or 6.4x depending on another. Accelerate Bench pairs executable workloads with agents' programs and trajectories to evaluate both their gains and the process behind them.

Read the original paper

More in AI Benchmarks

Browse all 60 papers →
02Benchmark

PReMind: When Personal Memory Meets Visual Hazards

Yeongtak Oh, Donghyeok Lee, Joonhee Oh, Jaewon Jeong, Sungroh Yoon

PReMind tests and trains visual assistants to connect what they see with the personal context needed to give safer advice.

Read analysis