ScienceIDE: Turning World's Scientific Codebase into Agent Learnable Environments

ScienceIDE converts scientific software repositories into agent-learnable environments by packaging pinned code, runtimes, official tests, expert-calibrated numerical checks, editable workspaces, and private verifiers. Its task factories generate repair, implementation, reproduction, acceleration, calibration, integration, and discovery challenges, but admit them only after execution proves that objectives are solvable, scientifically observable, and resistant to leakage. The current registry spans 64 environments across 27 codebases and contains 2,812 manufactured tasks, while a companion collection provides executable checks for numerical outputs and physical invariants. On the 85-task ScienceIDE-Hard benchmark with a one-hour budget, Anthropic’s Claude Fable 5.1 achieved 67.1% strict scientific success, ahead of OpenAI’s GPT-6 Astra at 63.1%; Claude Code, Codex, and Gemini CLI were used as execution harnesses, alongside models from Qwen and DeepSeek. Verified trajectories support supervised fine-tuning and reinforcement learning, including the PhAI-IDE-72B, PhAI-IDE-9B, and PhAI-IDE-4B family. Fine-tuning Qwen3.5-9B raised BBH Word Sorting from 0.240 to 0.576, suggesting transfer beyond scientific repair into reasoning and code benchmarks. For online learning, verifier-derived rewards and a truncation mask increased held-out LAPS reward from 0.357 to 0.857 and MITgcm-biogeo reward from 0.286 to 0.571, while preventing reinforcement learning from favoring prematurely shortened trajectories. The main limitation is coverage: most tasks concern reference-verifiable repair and implementation in computational physics and geoscience, not open-ended discovery or unseen codebases.