Are LLMs Ready for Scientific Discovery? A Capability-Oriented Benchmark for AI Scientists

The paper introduces SDABench, a capability-oriented benchmark that evaluates whether large language models can support scientific claims rather than merely execute analytical workflows. It organizes evaluation around six capabilities—descriptive, exploratory, inferential, predictive, causal, and mechanistic—across Biology, Chemistry, Environment, Geography, and Physics. The benchmark contains 527 real-data instances and 6000 synthetic instances, presented in both multiple-choice and open-ended formats, and evaluates 15 LLMs, including OpenAI’s GPT-5.4, Anthropic’s Claude Sonnet 4.6, Google DeepMind’s Gemini 3.1 Pro, DeepSeek, Qwen, GLM, and Meta’s Llama models. Its construction pipeline combines GPT-4o template generation, semantic causal graphs, programmatic ground-truth derivation, perturbation-based distractors, and human validation. On open-ended synthetic tasks, GPT-5.4 leads with 58.33% overall accuracy, but performance falls sharply beyond descriptive analysis: Environment is the strongest domain at 46.9% overall, while Biology reaches only 22.5% on predictive tasks. A five-stage error taxonomy—Scope, Variable, Function, Relationship, and Conclusion—shows that model scaling reduces early grounding errors, whereas later failures persist: Function Errors peak at 37.7% in predictive tasks, and Relationship Errors dominate mechanistic and exploratory reasoning. The central conclusion is that frontier LLMs can characterize observed data, but they remain unreliable at selecting assumptions, modeling variable relationships, generalizing to unseen cases, and constructing evidence-supported mechanisms.