ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research

ResearchClawBench, from Shanghai Artificial Intelligence Laboratory, is a benchmark for end-to-end autonomous scientific research that converts 40 real papers into executable tasks across 10 domains, including Astronomy, Chemistry, Earth Science, Energy, Information Science, Life Science, Materials, Mathematics, Neuroscience, and Physics. Each task hides the target paper and evaluates systems with expert-curated multimodal rubrics over the full research loop: reading literature, analyzing raw data, running code, and assembling a final report. The benchmark sets 50 points as the target-paper re-discovery boundary, so scores above 50 indicate discovery beyond the source paper. Under a unified protocol, the strongest autonomous agent, Claude Code, averages 21.5, while the best ResearchHarness LLM, Claude-Opus-4.7, averages 20.7; the LLM frontier mean is 26.5, and even the best autonomous-agent frontier mean is only 24.6. Error analysis over 280 runs shows failures concentrate in experiment-design mismatch, evidence mismatch, and missing scientific core, rather than simple execution failure. The paper also situates ResearchClawBench against systems and products from Anthropic, OpenAI, DeepSeek, Google, xAI, Qwen, Kimi, MiMo, and others, showing that current frontier models and agent scaffolds still fall far short of reliable target-paper-level scientific re-discovery.