Learning to Move Before Learning to Do: Task-Agnostic pretraining for VLAs

This Fudan University paper introduces Task-Agnostic Pretraining, or TAP, for vision-language-action models by separating “how to move” from “what to do.” Instead of treating every robot trajectory as a language-labeled expert demonstration, TAP first pretrains a Qwen2.5-VL 3B backbone with a SigLIP visual encoder on cheap unlabeled interaction data, including discarded Bridge trajectories and autonomous random play, using a self-supervised inverse dynamics objective that predicts the action between two observations. The second stage then fine-tunes on a small expert set with standard behavior cloning. On the SIMPLER benchmark, TAP-20k reaches 33.32% Avg-All success, outperforming Standard BC at 23.15% by 10% absolute and approaching internet-scale baselines pretrained on Open X-Embodiment, while using only 5k expert trajectories in stage 2. In real-world WidowX 250s experiments with only 200 expert demonstrations, TAP is pretrained on 30 hours of self-exploration and achieves strong robustness: under background texture shift it scores 25% on “put the carrot on the plate” versus 10% for NORA, and under viewpoint variation it still reaches 15% and 25% where NORA collapses to 0% on both tasks. The ablation studies show that scaling stage-1 exposure from 20k to 100k steps raises Avg-All from 18% to above 30%, and Grad-CAM visualizations confirm that inverse dynamics pretraining concentrates attention on manipulable objects before any language instruction is introduced.