ELA: End-to-End Latent Action Policy Learning from Video Demonstrations
AuthorsCheng-Chun Hsu, Peter Stone
AffiliationsDepartment of Computer Science, The University of Texas at Austin
ELA turns plentiful robot videos into useful control experience by jointly learning latent actions, policies, and action decoders.
Key results
Lightweight decoder overhead when integrating ELA with existing policies.
ELA-UWM result, compared with UWM at 19%.
ELA with Diffusion Policy, versus 0.48 for Diffusion Policy alone.
Ablation result, down from 0.83 with alignment.
ELA's action-prediction error using a fixed linear probe.
What the paper found
ELA tackles the gap between abundant action-free videos and scarce robot-control labels by jointly learning a latent action model, a latent-action policy, and an action decoder. An inverse dynamics model infers actions from video transitions, while a forward dynamics model grounds those representations in predicted visual change; iFSQ quantization regularizes the latent space. To prevent collapse and make latent actions both predictable and executable, ELA alternates updates and aligns policy features with frozen DINOv3 representations. Its decoder adds just 4M parameters and can extend existing policies without changing their architecture or training objective. On LIBERO-10, with 5 action-labeled demonstrations per task, ELA-UWM reaches 59% success versus 19% for UWM. In real-world tests using 10 labeled robot demonstrations and 40 human videos per task, ELA with Diffusion Policy achieves 0.70 mean success, compared with 0.48 for Diffusion Policy alone. Removing representation alignment reduces success from 0.83 to 0.45 in the reported ablation, while a linear probe achieves 0.015 action-prediction MSE, supporting the claim that ELA learns more usable action representations.
Original abstract
Robot imitation learning is limited by the scarcity of action-labeled data, while action-free demonstrations such as videos are abundant. Recent latent-action methods learn action-relevant representations from observation transitions, but they do not optimize latent actions for downstream control, leading to latent representations that are difficult for policies to model and difficult to accurately decode into continuous robot actions. This paper introduces ELA, an end-to-end framework for learning from video demonstrations via latent actions jointly optimized for policy learning and action decoding. ELA jointly trains a latent action model, latent action policy, and action decoder, enabling policies to leverage abundant video data with only a handful of action-labeled robot demonstrations. Experiments on simulated and real-world manipulation tasks show that ELA consistently outperforms prior latent action baselines. Moreover, it integrates seamlessly with existing imitation learning methods through lightweight 4M-parameter action decoders, requiring no changes to policy architectures or training objectives. Additional materials can be found on: https://end2end-latent-action.github.io/
Read the original paperMore in Robotics
Browse all 68 papers →GRASP: Gradient-Guided Sampling for Contact-Rich Physics-Based Motion Retargeting
Yang Liu, Tong Wang, Yiming Bao, Heng Yang
GRASP uses past simulation rollouts to guide robot motion retargeting, producing contact-rich movements that work in simulation and on real hardware.
PERA: Plug-in Experts for Robotic Actions in Continual Whole-Body Control
Shangchao Hu, Yadan Sun, Xibo Yu, Wei Huang
PERA teaches humanoid robots new skills by plugging small experts into an existing controller without disrupting its original motions.
Alloy: A Multi-Material Physics Engine for Robot Learning
Pranay Gupta, Deepanshu Rohilla
Alloy brings robots, cloth, cables, liquids, and other materials into one GPU-capable simulator for robot learning.