SafeDriveVLA: Navigation-Conditioned World Model Dreaming for Conflict-Aware End-to-End Autonomous Driving
AuthorsShaoyuan Xie, Zihan Zhang, Jingxuan Wang, Jiashu Qu, Xiaoqing Liang, Lingdong Kong, Junchi Lu, Henrik I. Christensen, Qi Alfred Chen
AffiliationsUC San Diego · University of Cincinnati · National University of Singapore Project Page: https://safedrive-vla.github.io/SafeDriveVLA
SafeDriveVLA imagines what would happen if it followed a driving instruction, helping it reject unsafe directions while navigating more reliably.
Key results
SafeDriveVLA's score on the closed-loop Bench2Drive benchmark.
SafeDriveVLA's success rate on Bench2Drive.
Navigation Compliance Rate, a 49.3% relative gain over the best SimLingo variant.
SafeDriveVLA collisions, compared with 60 for SimLingo-Safe.
Mean speed error in m/s; action anchors do not encode target speed.
What the paper found
As Li Auto deploys VLA driving and NVIDIA brings Alpamayo to the Mercedes-Benz CLA, SafeDriveVLA addresses a key safety problem: a navigation instruction can conflict with what the car sees. Its benchmark suite tests instruction following, scene conflicts, and adversarial instructions across CARLA-F, B2D-C, NavSim-C, and B2D-Adv. The model uses an InternVL3-1B backbone and a frozen V-JEPA 2 latent world model that imagines the scene under strict execution of the requested maneuver. It predicts one of three modes—strict, cautious, or fallback—before generating actions, separating the decision to follow an instruction from the driving action itself. On Bench2Drive, SafeDriveVLA achieves 88.16 Driving Score and 68.64% Success Rate. On CARLA-F, its language-following Navigation Compliance Rate reaches 83.0%, a 49.3% relative gain over the best SimLingo variant; on B2D-C, it reduces collisions from 60 to 38 compared with SimLingo-Safe. The ablations show the two components play complementary roles: mode prediction improves safety, while navigation-conditioned dreaming helps the model follow instructions when they are safe. A limitation is that the imagined action anchors do not encode target speed, contributing to a 4.28 m/s speed error on CARLA-F.
Original abstract
Vision-Language-Action (VLA) models are gaining attention in autonomous driving, inheriting commonsense priors from large language models (LLMs) and widening the navigation interface from a fixed command set to free-form natural-language instructions. However, this open-ended interface exposes a new vulnerability surface: an inattentive driver may issue instructions that are unsafe given the current scene, and a malicious user may issue dangerous ones on purpose. We present the first benchmark suite that evaluates navigation following and conflict awareness together, along with adversarial robustness. Existing VLAs fall short on both: SimLingo completes no language-instructed lane change in closed loop, and all models degrade sharply under conflicting signals. We propose SafeDriveVLA, the first driving VLA that (1) explicitly predicts a three-level navigation-compliance mode to decouple conflict reasoning from action generation, and (2) uses navigation-conditioned world model dreaming to inform this prediction. Unlike prior world models that roll out expert or planned actions, ours dreams of strictly following the navigation signal, showing when following is safe. SafeDriveVLA reaches 88.16 Driving Score (DS) and 68.64% Success Rate (SR) on Bench2Drive, lifts closed-loop language following by 49.3% and reduces collisions under unsafe instructions from 60 to 38 compared to the best SimLingo variant. Video demonstrations are available on our project page.
Read the original paperMore in Embodied AI
Browse all 54 papers →In-context Robot Learning Made Simple: A Democratized Recipe for Manipulation Tasks
Minxing Li, Minghao Han, Weizhi Zhao, Hanwen Wang, Xiangshuo Liu, Shuyao Shang, Jingxiang Zhou, Mingchao Sun, Hongyu Pan, Mu Xu, Yu Liu, Lue Fan, Zhaoxiang Zhang
SimpleICL presents an accessible recipe for teaching robots manipulation tasks from visual demonstrations without massive pretraining or specialized infrastructure.
NavGPT-3: Harnessing Context in a Hierarchical Navigation Runtime
Gengze Zhou, Yicong Hong, Jiazhao Zhang, Xunyi Zhao, Jian Zhou, Zixing Lei, Zun Wang, Chongyang Zhao, Xionghui Chen, Stephen Gould, Anton van den Hengel, Qi Wu
NavGPT-3 pairs a reasoning language model with a fast action policy so a robot can navigate complex routes at human-level performance.
Reflex: Visually Grounded Reactive Humanoid Control
Taoyang Jia, Weikai Huang, Linxin Song, Jared Darlington, Jieyu Zhao, Yue Wang, Jiafei Duan, Zhongzheng Ren, Ranjay Krishna
Reflex teaches a humanoid to use visual history to anticipate and catch thrown boxes, both in simulation and on a real robot.