Morphometric Imitation: From Morphology and Contact Aware Hand Retargeting to Sim-to-Real Visuomotor Policy

Morphometric Imitation converts reconstructed human hand-object interactions into zero-shot robot manipulation through three stages: Morphometric Optimization, or MMO, first scales the MANO hand model to match robot morphology and then recovers demonstrated contacts; residual reinforcement learning in ManiSkill adds dynamically feasible motion using object pose, contact state, PPO, and collision-aware termination; finally, demonstrations train a point-cloud visuomotor student with ManiFlow’s DiT-X action generator and flow-matching objectives. Across 10 GRAB trajectories and three robot hands—Dex3, Allegro, and Sharpa—MMO improves location-aware contact F1 over the strongest baseline by 27.8 percentage points on Dex3, while reducing contact patch error. The resulting residual policies reach success rates of 82.5% on Dex3, 69.9% on Allegro, and 91.8% on Sharpa. MMO runs at 180 frames per second on an NVIDIA RTX 4090, excluding inverse kinematics. For sim-to-real transfer, policies trained with randomized object scale, mass, friction, pose, and point-cloud observations achieve 89.3% success across 300 real-world trials involving 30 objects from 10 categories, without real-world policy training. The central finding is that morphology alignment and spatially precise contact preservation improve both dynamic feasibility and visual policy transfer, while object pose information guides task motion and contact information preserves the demonstrated grasp.