TCP4D: A Unified Tool-Center-Point Tracking and Generation Model for Robot Manipulation
AuthorsHaosong Peng, Xiangyang Liu, Zhengshen Zhang, Yunfeng Fan, Tianyu Qi, Xin Xia, Yufeng Zhan, Wenchao Xu
AffiliationsThe Hong Kong University of Science and Technology · Beijing Institute of Technology · National University of Singapore · The Hong Kong Polytechnic University · Sun Yat-sen University · Zhejiang University Project page: https://livioni.github.io/TCP4D/ Panda Hand Reconstruct
TCP4D turns robot videos into reusable motion demonstrations and uses them to guide manipulation across different robot embodiments.
Key results
Millimeters; TCP tracking benchmark result.
Millimeters; tracking on unseen in-the-wild tasks.
Millimeters; absolute trajectory error after adaptation.
Mean task success under clean2clean evaluation.
Average across placing pens and stacking bowls.
Percentage-point increase over augmented-only training.
What the paper found
TCP4D unifies robot tool-center-point tracking and action generation using a shared 4D scene representation. From monocular RGB video and an initial 2D query, it reconstructs metric scene geometry and camera parameters while directly estimating the TCP’s full SE(3) pose and gripper state—avoiding the rigid-point fitting and post-processing used by conventional trackers. Its generator samples visual features along tracked TCP paths and uses a Diffusion Transformer with flow matching, robot state, wrist-camera images, and language instructions to predict future motion. The visual backbone is based on Meta’s DINOv2, with T5-base encoding instructions. For cross-embodiment augmentation, TCP4D recovers motion from RGB, then uses inverse kinematics and robot rendering to create target-robot demonstrations without recorded source-robot states. Tracking achieves an absolute trajectory error of 7.60 mm on DROID and 16.07 mm on RoboLab; after adaptation with 10 demonstrations, error on a new UR5 embodiment is 8.85 mm. The generator reaches 69.19% mean success on RoboTwin and 78.33% across two real-world tasks, outperforming π0’s 45.21% RoboTwin score in the reported comparison. Joint training with source and augmented data raises mean ARX-X5 success from 30.5% to 72.0%, a 41.5-percentage-point gain.
Original abstract
Robot manipulation videos provide a rich visual experience, yet recovering robot action and gripper-state annotations from RGB observations still remains challenging. Using the embodiment-agnostic tool-center-point (TCP) pose as the action space, we present TCP4D, a unified model in which a shared 4D scene representation both tracks observed TCP motion and generates future TCP trajectories. Given a monocular video and initial 2D TCP queries, the tracker directly recovers metric TCP poses in SE(3) and gripper states alongside scene geometry and camera parameters. The generator samples local features from the same 4D representation along the recovered TCP tracks, then combines them with robot observations and language instructions to predict future TCP states. Our new cross-embodiment augmentation pipeline converts existing RGB videos into demonstrations for alternative embodiments by retargeting the recovered motion and synthesizing the corresponding observations, without recorded source-robot states. Experiments show that the tracker achieves absolute trajectory errors of 7.60 and 16.07 mm on DROID and RoboLab and adapts to a new embodiment with only ten demonstrations. The generator achieves 69.19% mean success on RoboTwin and outperforms well-established policy models on two real-world tasks, while joint training on source and augmented demonstrations improves target-robot success by 41.5% over augmented-only training.
Read the original paperMore in Embodied AI
Browse all 69 papers →Demonstration of MANA: Multi-perspective Alignment of NPC Agents
Amir Tahmasbi, Keshav Singh, Shaheer Raza, Owen Mei, Jason Wu
MANA lets game developers step into an NPC’s shoes and demonstrate how it should behave, then lets the character act on its own.
Kimodo-Policy: Learning Kinematic Action Models for Humanoid Loco-Manipulation
Yunheng Wang, Jiaxi Zhang, Wei Lu, Yuetong Fang, Taowen Wang, Yixiao Feng, Shuning Zhang, Junzhe Xu, Yichi Wang, Zizhao Yuan, Kailun Huang, Lusong Li, Mingqian Zhou, Zecui Zeng, Renjing Xu
Kimodo-Policy turns a pretrained whole-body motion generator into a control policy for humanoid robots that walk and manipulate objects.
One Policy for 100 Household Tasks: Our Solution for the 2026 BEHAVIOR Challenge
Ilia Larchenko, Gleb Zarin, Atindra Nair, Keerti Manney, Vitaly Kleban, Mirko Kiefer, Anjana Yodaiken
A single vision-language-action policy tackles 100 simulated household tasks, with a coding agent stepping in for some of the hardest ones.