Scaling Online Exploration for Robot Agents
AuthorsRunhan Huang, Xiangcheng Zhang, Haoyu Chen, Yitong Li, Yen-Jen Wang, Bohan Lyu, Haotian Ye, Chuye Hong, Yixuan Wang, Chloe Su, Haoyu Xiong, Yangzhen Wu, Enze Xie, Travis Manderson, Stefano Ermon, Pieter Abbeel, Antonio Torralba, Dawn Song, Weirui Ye, Yilun Du
Affiliations[
RAX lets coding agents explore, learn, and compose robot skills across different embodiments without relying on human demonstrations.
Key results
Percentage points above the strongest prior models on each benchmark.
Average after 32 exploration episodes, without action demonstrations.
Average after at most five adaptation episodes; VPP2 scores 42.3%.
Average using three teleoperated demonstrations per task; the strongest comparison model uses 100.
Millimeters of clearance in a real-robot precision insertion task.
What the paper found
RAX changes robot learning from copying demonstrations to learning through online exploration: a frozen coding agent interacts with robot sensors and controls, tests actions, and turns successful strategies into executable Python programs for perception, planning, and control. Verified skills are saved and reused, so the system can adapt to new scenes without updating model weights or relying on hand-built action primitives. The study uses OpenAI Codex and Claude Code as agents, with Google Gemini for visual grounding, and compares RAX with vision-language-action models including OpenVLA and π0.5. Across five simulation benchmarks, RAX beats the strongest prior models by an average of 37.4 percentage points; on DexJoCo it reaches 96.7% success after 32 exploration episodes, without action demonstrations. On LIBERO-Pro, five or fewer adaptation episodes raise its average success to 84.4%, compared with 42.3% for VPP2. HumanoidArena results reach 89.8% using three teleoperated demonstrations per task, while the strongest comparison model uses 100. On real robots, RAX also handles tasks such as inserting pegs into holes with 0.1 mm clearance, and composes previously learned skills into longer household tasks. The central finding is that scaling an agent’s experience—and compiling what it learns into reusable code—can improve generalization and reduce reliance on robot-specific demonstration datasets.
Original abstract
Robot learning has advanced by scaling demonstration datasets and model capacity; however, deploying a new robot embodiment or task typically necessitates a dedicated teleoperation corpus. The resulting policies remain bounded by their imitation distributions, limiting generalization and continual self-improvement. We introduce RAX (Robot Agent eXploration), an open-source framework for robot agents centered on scaling exploration: enabling agents to improve by acquiring online experience and synthesizing executable programs. In RAX, a general-purpose coding agent generates robot control code, operating without action demonstration data, hand-engineered action primitives, or value-function guidance. Through online exploration, the agent identifies decision sequences that achieve task goals and compiles them into modular programs encapsulating the full perception-planning-control loop. Using RAX, we present a systematic multi-embodiment study on scaling online exploration across precision and long-horizon manipulation tasks using single-arm, bimanual, dexterous, mobile, and humanoid systems in simulation and on real robots. Generalization performance on unseen environments improves consistently as exploration scales. In simulation, RAX outperforms top baseline models by +37.4% on average. On real robots, RAX autonomously learns memory-dependent, articulated, high-precision, deformable, and long-horizon mobile manipulation tasks without human demonstrations, continually improving performance by composing verified skills from prior exploration sessions. As exploration scales, we observe emergent capabilities including active perception, error recovery, safety guardrails, and human-robot collaboration. We open-source RAX as a generalist robot-agent platform in which new robot embodiments are integrated through their APIs alone, without teleoperated data collection or policy training.
Read the original paperMore in Embodied AI
Browse all 57 papers →Self-Improving Video-to-Action Models
Hyunwoo Ryu, Eric Ming Chen, Max Wilcoxson, Vincent Sitzmann
Mirror helps embodied agents turn visual plans into actions by learning from their own attempts in the environment.
SuperMimic: Scaling Humanoid Whole-Body Control with One Differentiable World Model
Qiayuan Liao, Yiyang Shao, Junfeng Long, Kevin Zakka, Roberto Horowitz, Pieter Abbeel, Koushil Sreenath
SuperMimic uses a differentiable world model to train humanoid motion controllers far faster, and demonstrates the result on a real robot.
In-context Robot Learning Made Simple: A Democratized Recipe for Manipulation Tasks
Minxing Li, Minghao Han, Weizhi Zhao, Hanwen Wang, Xiangshuo Liu, Shuyao Shang, Jingxiang Zhou, Mingchao Sun, Hongyu Pan, Mu Xu, Yu Liu, Lue Fan, Zhaoxiang Zhang
SimpleICL presents an accessible recipe for teaching robots manipulation tasks from visual demonstrations without massive pretraining or specialized infrastructure.