Adaptive Vision-Language Grasping via Composable Foundation Priors and Generalizable Grasp Synthesis

AdaRoboVLG addresses the brittleness of end-to-end vision-language grasping by separating task understanding from physical grasp synthesis. Its structured grasp interface combines object geometry, Contact Grasp Representations encoding contact point, approach and closing directions, grasp width, and taxonomy-compatible grasp types. A hand-agnostic base policy maps these primitives to executable poses for different robotic hands, then uses a PointTransformer-based hand-object interaction model and force-closure stability estimation to rank candidates. Three composable priors modify the same interface: a spatial prior uses DINOv3 features, multi-view RGB-D, and Contact-GraspNet to generate clutter-aware contact constraints; a cognitive prior combines ByteDance’s Seed-1.8, Retrieval-Augmented Generation, Chain-of-Thought reasoning, Qwen3.5-VL-generated instructions, and SAM3 grounding to infer functional parts and grasp types; and a temporal prior fuses SAM3 mask tracking with DINOv3 cross-frame features for rigid-motion updates. On DexGraspNet 2.0, spatial reasoning reaches an average 88.4% success rate across three hands and clutter levels. Functional inference reaches 94.0% functional-part accuracy and 87.0% grasp-type accuracy, while language-guided grasping achieves 81.3% on GraspClutter6D and 86.0% on GraspNet-1Billion. The policy is trained from 4.4M simulated grasp trials and transfers across two- to five-finger hands. In real-world tests using NVIDIA hardware, it achieves 83.3% success across 510 trials and 89.7% under human-induced dynamic disturbances, with online updates at 5 Hz. The key result is modularity: new perception or reasoning capabilities can be added without retraining the physical grasp policy.