In-Context World Modeling for Robotic Control

In Context World Modeling for Robotic Control, from Fudan University and the Shanghai Innovation Institute, reframes generalization failure in Vision-Language-Action systems as test-time system identification: instead of assuming a fixed camera pose or robot morphology, the policy first collects a short self-generated interaction prefix and uses that context to infer the latent configuration before acting. The method, ICWM, reuses a Qwen2.5-VL-3B backbone with FAST action tokenization and requires no parameter updates, task-specific demonstrations, or reward signal at deployment. On LIBERO with 8 training viewpoints and 6 unseen OOD viewpoints, ICWM raises average unseen-view success from 19.8% with Multi-View BC to 25.0%, and the paper reports especially large gains on LIBERO-Long, where OOD success reaches 25.0% versus 19.8%. In real-robot experiments on a 6-DoF UR5e with 12 cameras, standard multi-view performance drops from 68% to 17% under viewpoint shift, while ICWM substantially narrows that gap. Ablations show the interaction prefix matters structurally: removing images causes the largest collapse, and false context performs worse than no context, indicating the model genuinely conditions on inferred world dynamics. The approach also transfers beyond camera shifts, improving under distractor objects, novel table textures, and morphology changes such as rigid spacers of 20, 40, and 80 mm, with a particularly notable recovery at 80 mm where the baseline largely fails.