Ego2Robot: Scalable Robot Data Synthesis from Egocentric Human Data

Ego2Robot, from researchers at Renmin University of China, Alibaba’s Qwen Team, ShanghaiTech, and BIGAI, presents a pipeline for turning egocentric human manipulation video into robot-training data at scale. It estimates hand motion with WiLoR and DynHaMR, retargets trajectories to robot grippers, searches feasible robot bases with MuJoCo inverse kinematics, removes human arms using SAM 3 and ProPainter, composites rendered robots with depth, and applies trajectory, statistical, and Qwen3.5 vision-language consistency filtering. From 1,940 hours across ANT, EgoDex, ViTRA, and EgoVerse, the system produces 18,561 hours across 15 robot morphologies. Using a Qwen3.5-4B vision-language-action model with a Diffusion Transformer action head, the authors extend RoboTwin2.0 to isolate visual, scene, embodiment, and task-semantic shifts. Mixing Ego2Robot and real robot data at 1:1 reaches 53.5% on RoboTwin Randomized, 2.6 percentage points above robot-only pretraining, while the 3:1 mix reaches 51.7% on EBench, a 12.1-point gain. Improvements are strongest for lighting, camera offsets, cross-embodiment transfer, unseen objects, and paraphrased instructions. In an ARX ACone deployment, adding converted egocentric play data improves five long-horizon tasks, with gains of 14 points on Put Blocks and 13 points on Insert Screw, suggesting that synthesized human demonstrations complement rather than replace real robot data.