NTH
Robotics

Dex-One2Many: Learning Dexterous Manipulation from a Single Human Demonstration

AuthorsJusuk Lee, Sungha Kim, Yeonsoo Park, Jonguk Cheon, Yoonkyo Jung, Yongjun You, H. Jin Kim, Jia-Bin Huang, Furong Huang, Youngseok Jang, Seungjae Lee

AffiliationsSeoul National University University of Maryland, College Park · All Purpose AI

October 17, 2026 2 min read
Watch on YouTube

Resources

View on Hugging Face Read the original paper on arXiv
The one-line take

A single human video is transformed into scene-graph guidance that helps a robot hand learn dexterous skills and adapt to situations it never saw.

Key results

75%
Unseen-configuration mean success

Across five tasks; the best baseline averaged 4%, a 71-percentage-point gap.

60–85%
Real-world unseen success range

Success across the five real-robot tasks.

94.6–96.8%
Simulation success across embodiments

Mean task success across four robot-hand embodiments.

49,152
Parallel training environments

Used for PPO training; smaller-scale runs stayed below 25% success within the same updates.

What the paper found

Dex-One2Many turns one human manipulation video into a policy that learns the task, not a replay of the demonstrated motion. Gemini 3.1 Pro extracts a sequence of scene graphs describing relations such as grasp, contact, and placing an object inside a container; OpenAI’s GPT-5 helps estimate object physics. The graphs leave poses and grasps flexible, while generating diverse simulation reset states and dense, stage-specific rewards for PPO. Dexonomy supplies varied multi-finger grasps, and a QP controller converts the learned hand and palm commands into safe joint motion. Trained in NVIDIA Isaac Sim, the policy transfers zero-shot to a real robot. Across five tasks, its average success on unseen configurations is 75%, versus 4% for the best baseline—an advantage of 71 percentage points—and real-world success ranges from 60–85%. In simulation, success averages 94.6–96.8% across four different robot-hand embodiments. Training uses 49,152 parallel environments; ablations show that removing intermediate-stage resets or dense predicate rewards sharply undermines learning. The central advance is using a single video to specify what must happen and in what order, while reinforcement learning discovers how to do it across new poses and grasps.

Original abstract

While learning dexterous manipulation from a single human video offers a promising alternative to costly robot demonstrations, many recent methods predominantly imitate demonstrated motions. Such strict motion matching often limits generalization to initial object poses, goal poses, and grasps not shown in the video. Alternatively, discovering a policy via reinforcement learning (RL) allows for broad generalization, but without prior guidance, it struggles with high-dimensional exploration in complex, multi-stage tasks. To address these coupled generalization and exploration challenges, we present Dex-One2Many, a real-to-sim-to-real framework that learns a generalizable dexterous manipulation policy from a single human video. Our key insight is to abstract the video into sequential scene graphs that guide RL, enabling efficient exploration while preserving broad generalizability. The graphs serve as generative constraints for sampling diverse reset states and provide dense rewards for each stage. Because the graphs constrain relations rather than exact poses, these reset states cover object poses and grasps beyond the video, while initializing each stage from them with dense rewards keeps exploration short and guided. Trained entirely in simulation, Dex-One2Many transfers zero-shot to a real multi-fingered hand. Across five tool-use and manipulation tasks, Dex-One2Many exceeds baselines by 6.5% in seen configurations, while its robust generalization widens this gap to 71% in unseen scenarios.

Read the original paper

More in Robotics

Browse all 53 papers →