NTH

ELA: End-to-End Latent Action Policy Learning from Video Demonstrations

AuthorsCheng-Chun Hsu, Peter Stone

AffiliationsDepartment of Computer Science, The University of Texas at Austin

November 18, 2026 2 min read
Watch on YouTube
The one-line take

ELA turns plentiful robot videos into useful control experience by jointly learning latent actions, policies, and action decoders.

Key results

4M
Action decoder parameters

Lightweight decoder overhead when integrating ELA with existing policies.

59%
LIBERO-10 success with 5 labeled demonstrations

ELA-UWM result, compared with UWM at 19%.

0.70
Real-world mean success

ELA with Diffusion Policy, versus 0.48 for Diffusion Policy alone.

0.45
Success without representation alignment

Ablation result, down from 0.83 with alignment.

0.015
Linear-probe action-prediction MSE

ELA's action-prediction error using a fixed linear probe.

What the paper found

ELA tackles the gap between abundant action-free videos and scarce robot-control labels by jointly learning a latent action model, a latent-action policy, and an action decoder. An inverse dynamics model infers actions from video transitions, while a forward dynamics model grounds those representations in predicted visual change; iFSQ quantization regularizes the latent space. To prevent collapse and make latent actions both predictable and executable, ELA alternates updates and aligns policy features with frozen DINOv3 representations. Its decoder adds just 4M parameters and can extend existing policies without changing their architecture or training objective. On LIBERO-10, with 5 action-labeled demonstrations per task, ELA-UWM reaches 59% success versus 19% for UWM. In real-world tests using 10 labeled robot demonstrations and 40 human videos per task, ELA with Diffusion Policy achieves 0.70 mean success, compared with 0.48 for Diffusion Policy alone. Removing representation alignment reduces success from 0.83 to 0.45 in the reported ablation, while a linear probe achieves 0.015 action-prediction MSE, supporting the claim that ELA learns more usable action representations.

Original abstract

Robot imitation learning is limited by the scarcity of action-labeled data, while action-free demonstrations such as videos are abundant. Recent latent-action methods learn action-relevant representations from observation transitions, but they do not optimize latent actions for downstream control, leading to latent representations that are difficult for policies to model and difficult to accurately decode into continuous robot actions. This paper introduces ELA, an end-to-end framework for learning from video demonstrations via latent actions jointly optimized for policy learning and action decoding. ELA jointly trains a latent action model, latent action policy, and action decoder, enabling policies to leverage abundant video data with only a handful of action-labeled robot demonstrations. Experiments on simulated and real-world manipulation tasks show that ELA consistently outperforms prior latent action baselines. Moreover, it integrates seamlessly with existing imitation learning methods through lightweight 4M-parameter action decoders, requiring no changes to policy architectures or training objectives. Additional materials can be found on: https://end2end-latent-action.github.io/

Read the original paper

More in Robotics

Browse all 68 papers →