Human Universal Grasping

Human Universal Grasping, or HUG, from New York University, Tsinghua University, the University of Michigan, and Meta Reality Labs’ Project Aria team, reframes dexterous grasp learning around human data instead of robot teleoperation or simulation. The system is trained on 1M-HUG S, a 1M-frame egocentric dataset collected with Aria Gen 2 smart glasses across 6,707 recordings and 41 buildings, then learns a flow-matching transformer that fuses RGB-D with a queried 3D object point to predict a 99-dimensional MANO grasp state: wrist translation, 6D wrist rotation, and 15 joint rotations. HUG-B ENCH standardizes evaluation with 90 unseen objects spanning five geometry classes and three size bins, reconstructed into metric-scale meshes for both simulation and real-robot trials. In MuJoCo, the full RGB+PC model reaches 73.0% success on the test split with 14.6 mm fingertip contact error, while an oracle replay of recorded human grasps reaches 94.0% and 7.4 mm. The key ablation shows that removing the 3D supervision drops test success to 32.7%, and removing RGB drops the model to 70.7%, confirming that geometry and semantics are complementary. In real-world tabletop trials on 30 unseen objects, HUG achieves 66.7% success, outperforming Dex1B at 43.7% and CAP at 32.7%, and it still reaches 62.0% in an uncontrolled in-the-wild household setting after zero-shot retargeting to the Ability Hand and WUJI Hand without per-hand training.