InSight: Self-Guided Skill Acquisition via Steerable VLAs

InSight, from Stanford University and Princeton University, reframes vision-language-action learning by making a policy steerable at the primitive level and then using a vision-language model, Gemini 3 Flash, to discover and acquire missing primitives without new human demonstrations. The system first auto-segments teleoperated data by aligning VLM-generated primitive plans with gripper transitions, end-effector motion, and visual cues, then fine-tunes a π0.5 VLA with LoRA and a learned progress channel. In the second stage, the VLM decomposes a novel task, flags primitive gaps, proposes single-axis parameters for each gap, and filters successful rollouts with oracle checks before retraining the VLA on the newly acquired primitives. On simulation block flipping in LIBERO, 150 pick-and-place demonstrations were segmented into over 700 primitive episodes, and InSight reached 75% success after 246 acquired primitive rollouts, while an SAC baseline never completed a flip under the same rollout budget. On hardware with a 6DoF UFactory xArm, InSight acquired 20 successful twist primitives in 23 trials and 20 successful pour primitives in 31 trials, then achieved 92% success on twisting, 96% on pouring, and 80% on a 14-primitive twist-then-pour composition, compared with 32%, 16%, and 4% for CaP-X and 0% on the π0.5 baseline. The unified policy retained 100% success on original pick-and-place skills, and InSight also learned sweeping from scooping demonstrations, succeeding in 5/5 trials.