Future Representation Distillation for Imagination-free World Action Models
AuthorsJaehyeong Jo, Sangwon Jang, Sihyun Yu, Seonghyeon Ye, Yekyung Nah, Hojung Jung, Kimin Lee, Sung Ju Hwang
AffiliationsKAIST
DreamFree teaches a robot to use a video model’s learned sense of the future without having to generate that future frame by frame.
Key results
Wan2.2 backbone used for the reported real-time policy.
Hz achieved with the 14B backbone on an NVIDIA H200.
Inference speedup without future-video generation.
Average success rate using the 14B backbone.
Average rate across 10 DROID tasks, versus 56% for DreamZero and 16% for π0.5.
What the paper found
DREAMFREE makes a video diffusion model’s knowledge of how scenes evolve useful for robot control without generating future videos at deployment. In a two-stage process, a visual encoder initialized from Wan2.2 learns from the current image and instruction to predict representations produced by a frozen teacher with access to future frames; a structured attention mask prevents the encoder from seeing those future frames. The encoder is trained with both future-representation distillation and the video model’s generative objective, then an action readout and a 1B-parameter action DiT map its features to robot actions. This also lets the visual encoder learn from videos without action labels. With a 14B backbone, DREAMFREE reaches 11Hz closed-loop control on an NVIDIA H200 and runs 17.8 times faster than DreamZero. It achieves 71.7% success on RoboCasa-Kitchen, and in real-world tests its 64% success rate exceeds DreamZero’s 56% and π0.5’s 16%. The results show that distilling predictive features can improve control while avoiding the cost and potential errors of future-video generation; video-only task demonstrations also enable transfer to tasks lacking action annotations, including across robot embodiments.
Original abstract
World Action Models (WAMs) use the predictive knowledge of pretrained video diffusion models for robot control through future video prediction. However, iterative video denoising is costly, and inaccurate visual predictions can compromise action prediction. Our key idea is to make this predictive knowledge available through visual representations computed from the current observation, enabling action prediction without generating future videos. We introduce DreamFree, a WAM that repurposes a pretrained video diffusion model as a predictive visual encoder through representation distillation. The visual encoder learns to predict a frozen video diffusion model's representation using only the current observation. An action module is then trained to predict actions by extracting action-relevant information from the visual encoder. Across real-world and simulated tasks, DreamFree achieves higher success rates than state-of-the-art VLAs and WAMs, while performing real-time closed-loop control at 11Hz with a 14B video diffusion backbone. Experiments show that DreamFree can translate stronger video priors and additional video data into better robot control and enable learning new robot tasks from video demonstrations, including videos from a different embodiment, without action annotation for the new tasks.
Read the original paperMore in World Models
Browse all 59 papers →Duo: A Reciprocal World Model for Full-Duplex Interaction
Hengyuan Cao, Biao Gong, Guoshan Lu, Qiyuan Zhang, Lin Liu, Yiding Tian, Yilei Jiang, Yuxin Ma, Chengzhang Wu, Kun Chen, Yiqian Xie, Ting Liu, Xin Zheng, Xijun Gu, Tianren Ma, Jiancheng Pan, Chi Zhang, Xuzhe Zheng, Siyi Li, Zhenzhong Lan
Duo aims to create an always-on audiovisual assistant that can perceive, reason, and respond continuously, including during interruptions.
PanDWM: Panoramic 4D Driving World Models
Jiajun Sun, Zeyu Zhang, Zeyu Ren, Guangfeng Jiang, Yang Zhao, Hao Tang
PanDWM generates a continuous 360-degree driving view and reconstructs it into a fast 4D scene.
WorldRuler: Interactive World Modeling under Configurable Rules
Mingliang Zhai, Chuanhao Li, Jiaming Tan, Zhen Li, Yuwei Wu, Yunde Jia, Zian Meng, Yuanyang Yin, Kaipeng Zhang
WorldRuler aims to make generated interactive worlds obey customizable rules, even after saving and restoring the world.