Right Answer, Wrong Method: Shortcut Hacking Misleads the Evaluation of LLM Reasoning on Frontier Science Benchmarks

Researchers at Alibaba Group and Alibaba DAMO Academy identify “solution hacking,” in which an LLM reaches the correct answer through numerical search, enumeration, guessing, formula recall, or answer-first verification instead of performing the reasoning capability a benchmark targets. Auditing 3528 solutions from frontier models including OpenAI’s GPT-5.2, Anthropic’s Claude Opus 4.7, Google DeepMind’s Gemini-3.1-Pro-Preview, and DeepSeek models, they show hacking rises with difficulty: from 2.2% on easy textbook problems to 28.3% on olympiad-level competition problems and 37.4% on Humanity’s Last Exam, or HLE. Among answers marked correct, hacked solutions account for 8.2%-44.1%, with weaker models showing the greatest inflation. The authors define derivation-adjusted accuracy, which credits only correct, non-hacked solutions, and build a majority-vote expert-anchored judge with 75.4% agreement against blinded PhD annotations; it detects 61.2% of expert-confirmed hacks, making reported rates conservative lower bounds. An anti-hacking instruction that permits abstention reduces hard-core reported accuracy from 41.5% to 33.3% while lowering the hack rate from 22.3% to 6.9% and leaving derivation-adjusted accuracy nearly stable. The central conclusion is that answer-only evaluation can substantially overestimate frontier LLM scientific reasoning, especially when problems have searchable or easily verifiable answer spaces.