Ask, Don't Equip: Question-Driven Coding Agents for Deep Kernel Optimization
AuthorsGabriele Oliaro, Jaeseong Lee, Zhaoyuan Su, Sreeram Vennam, Zhihao Jia, Samyam Rajbhandari
KernelSeek shows how asking an AI coding agent the right question can help it build faster GPU kernels without handing it the answer.
Key results
Geometric-mean speedup over the seed-relative FA3 benchmark after 133 model-driven steps.
Geometric-mean speedup over FlashAttention-3 on the same 26 configurations.
Geometric-mean speedup over FlashAttention-3 across 182 Hopper scenarios.
Geometric-mean speedup over FlashAttention-4 on paired Blackwell scenarios.
Scenarios on which the Blackwell kernel was correct.
Reusable questions designed to direct measurements without prescribing solutions.
What the paper found
KernelSeek replaces tool-heavy, autonomous kernel search with a general-purpose coding agent, a minimal harness that checks correctness against the seed and an fp32 reference, and short questions posed when progress stalls or the agent claims a limit. The questions provide no answers: they prompt measurements and redirect attention across the kernel, host-launch path, routing, and benchmark. A reusable bank contains 101 core questions, and the agent can apply them to itself. In a controlled comparison using Claude Opus on NVIDIA H200 hardware, a tool-rich autonomous agent plateaued at 0.50× FlashAttention-3 speed after 133 model-driven steps, while KernelSeek reached 1.13× on the same 26 configurations. In full campaigns, its dependency-free CUDA kernels—using techniques such as warp specialization, asynchronous tensor-core operations, and host-side routing—achieved 1.16× FlashAttention-3 on Hopper across 182 scenarios and 1.46× FlashAttention-4 on Blackwell; the Blackwell kernel was correct on 154 runnable scenarios. The trace analysis found that large gains followed questions that changed which system layer the agent examined or challenged an assumed ceiling, not questions that supplied new technical information. The results suggest that directing an agent’s measurements can matter more than adding tools or reference material.
Original abstract
LLM agents for GPU kernel optimization sit at two ends of a structure-freedom tradeoff. Scripted agents fix what the model does at each step, via a learned policy, an optimization menu, or fixed roles, and are sample-efficient where the recipe applies and stall where it does not. Open-ended agents let a self-directed model mutate a population of kernels and pay for the freedom in trials: thousands of evaluations, repeated edits, and wrong candidates. Both respond to hard targets by equipping the agent with more: parsed profiler output, disassembly analysis, retrieval over documentation and libraries, and stores of lessons. We argue that both lack direction, a hypothesis about where the remaining gap is, and that a short question asked at the right moment supplies it without narrowing what the agent may do next. We present KernelSeek, a question-driven agent that makes an existing CUDA kernel faster with no operator-specific knowledge and no faster implementation to imitate: a general-purpose coding agent, a minimal harness that enforces a verifiable goal, and questions at decision points, first from a human, then from a bank the agent applies to itself. On the same task, evaluator, hardware, and week, a tool-rich autonomous agent built in the prevailing style plateaued at 0.50x FlashAttention-3 after 133 model-driven steps; KernelSeek reached 1.13x. It then produced dependency-free FlashAttention forward kernels for Hopper and Blackwell, single CUDA files with no CUTLASS, CuTe, cuBLAS, or cuDNN, correct on all 182 scenarios per architecture and 1.16x FlashAttention-3 and 1.46x FlashAttention-4 (geometric mean). Across 162 human messages and 11.5K agent tool calls, every large gain followed a question that moved the agent to a different layer of the system or rejected a claimed limit, never one that supplied information. We release the kernels, harness, question bank, and all traces.
Read the original paperMore in Code Generation
Browse all 52 papers →CompoundingKernel: Self-Improving from Zero to First
Qian Qu, Hongyang Wang, Wenbo Chen, Yichuan Li, Jingguo Liu, Xian Li, Xuan Wang, Ruochen Wen, Leo Fan, Yiming Yan, Zheshen Wang, Benoit Dumoulin
CompoundingKernel uses parallel agents and an archive of successful attempts to discover fast, correct GPU kernels.
CoReMem: Cost-Internalizing Memory for Successive Coding Tasks with Prospective Valuation
Peiwen Li, Harshit Verma, Sophia Zhou, Haiwen Wang, Ngoc Bui, Heng Chang, Fei Fei, David van Dijk, Rex Ying
CoReMem helps coding agents carry forward useful repository knowledge while accounting for the cost of writing and using memory.
From Repositories to Training Environments: Scaling Long-Horizon Coding Agents with Environment Evolution Loop
Yang Tian, Tianyu Zheng, Xiyuan Yang, Yaxin Du, Chengran Yang, Bo Zhao, Jian Yang, Bryan Dai
RepoFactory turns real software repositories into training environments and iteratively improves them to help coding agents tackle longer, more realistic projects.