RoboColosseum: Evaluating Physical Intelligence at Scale In-The-Wild
AuthorsTan Sang Nguyen, Yi Ru Wang, Shailesh Shailesh, Gang Yang, Jiamo Hu, Jianuo Jiang, Haoquan Fang, Zhongzheng Ren, Zhongyi Luo, Ian Chong, Aaron Tung, Santiago Pravisani, Minghui Lin, Yao Mo, Ishneet Singh, Benjamin Tee, Harold Soh, Siddhartha Srinivasa, Dieter Fox, David Hsu, Haoming Song, Maxime Ellerbach, Thomas Wolf, Pepijn Kooijmans, Ruijie He, Michael Cho, Jiafei Duan
Affiliations[
RoboColosseum tests robot policies on real robots at scale, showing that success depends strongly on the task, embodiment, and whether policies are fine-tuned.
Key results
Blinded pairwise physical evaluations.
Sites contributing Open Track comparisons.
Tasks across four robot embodiments.
Shared adaptation data for each Fine-tune Track task.
Indicates weak agreement between policy rankings across embodiments.
Lower end of the 400–500 evaluation range needed for about 90% ranking agreement.
What the paper found
RoboColosseum is a distributed benchmark for testing robot intelligence on physical hardware across changing tasks and environments. Its Open Track compares policies without task-specific training, while its Fine-tune Track gives them the same 100 demonstrations per task; together, the tracks cover 20 tasks and four robot embodiments. In the Open Track, 5,699 blinded pairwise comparisons across 124 sites show that frontier AI agents—including OpenAI’s GPT-6 Astra, Anthropic’s Claude Opus 5.5, and Grok 4.7—can rival vision-language-action models such as NVIDIA’s GR00T N1.7 and π0.5, though reliable task completion remains difficult. Fine-tuning changes the picture: GR00T N1.7 leads on SO-101, while π0.5 leads on Franka and Bimanual-YAM, and rankings agree only weakly across embodiments, with Kendall’s W at 0.24. On the G1 humanoid, no policy completes any task, despite average progress ranging from 13.3% to 23.2%—evidence that success rates alone hide meaningful differences. To make costly physical evaluation more efficient, RoboColosseum uses active matchmaking to prioritize uncertain model comparisons; it reaches about 90% agreement with the full-data ranking after 400–500 evaluations, compared with roughly 1,000 under uniform sampling.
Original abstract
General-purpose robot policies must work across robots, tasks, and environments, yet the variation that makes physical trials informative also makes them difficult to reproduce. We introduce RoboColosseum, a distributed evaluation framework that combines unified policy serving and standardized protocols across a network of locally operated robots in diverse environments. Our initial study evaluates five vision-language-action (VLA) policies and three frontier AI agents across four embodiments (SO-101, Franka, Bimanual-YAM, and G1 Humanoid). The Open Track compares models without task-specific adaptation in blinded, pairwise trials under matched local conditions. The Fine-tune Track measures how well VLA policies adapt to 20 tasks across all four embodiments when each policy receives the same 100 demonstrations per task. Across 5,699 pairwise Open Track comparisons at 124 sites, frontier agents rival specialized VLA policies, while all systems struggle to complete open-world tasks reliably. After fine-tuning, GR00T N1.7 leads on SO-101 and pi0.5 leads on Franka and Bimanual-YAM. Policy rankings show weak agreement across embodiment-specific task suites (Kendall W = 0.24). Open Track rankings poorly predict relative performance after fine-tuning, with substantial rank reversals on all three embodiments shared between tracks. Measuring partial progress reveals differences that task success alone misses. On the G1 Humanoid, no policy completes any of the five tasks, yet average progress ranges from 13.3% to 23.2% across policies. Open Track rankings stabilize as trials accumulate, and active matchmaking achieves comparable ranking agreement with substantially fewer physical trials than uniform sampling. These results show how distributed physical evaluation can reveal where robot policies generalize, how well they adapt to new tasks, and how far they progress on tasks they fail to complete.
Read the original paperMore in AI Benchmarks
Browse all 60 papers →Accelerate Bench: Evaluating Agents for Application-Level Program Optimization
Bowen Liu, Yuhan Li, Nuo Chen, Weile Luo, Miao Peng, Qintong Zhang, Qifan Zhang, Jia Li, Ziqi Gao, Lydell Huang
A new benchmark tests whether coding agents can make whole scientific programs faster without breaking them.
DrawScore: Evaluating Generated CAD Models against Engineering Drawings
Haorui Chen, Shiteng Lu, Zhanyu Guo, Zaixi Li, Yunhe Ni, Yaosong Du, Yichen Xiang, Xiande Huang, Rui Huang
DrawScore checks generated CAD models through engineering-style views, making small but manufacturing-critical mistakes harder to overlook.
PReMind: When Personal Memory Meets Visual Hazards
Yeongtak Oh, Donghyeok Lee, Joonhee Oh, Jaewon Jeong, Sungroh Yoon
PReMind tests and trains visual assistants to connect what they see with the personal context needed to give safer advice.