Foundation4D: Towards a unified perception and reconstruction model for autonomous vehicles

NVIDIA’s Foundation4D combines driving perception and scene reconstruction in one feed-forward model. A shared DINOv2 ViT-L/14 backbone processes multi-camera video using factorized attention across views and time, while separate heads predict camera poses, metric depth, 3D objects, semantic occupancy, and renderable 3D Gaussians. For long recordings, pose-graph optimization aligns overlapping windows, and offline tracking links detections into object trajectories. The team also introduces PAI–Foundation4D, a benchmark with dense depth and occupancy annotations for 1,253 driving clips. After benchmark-specific fine-tuning, the model reaches 60.9 mAP on nuScenes and 18.1 mIoU on OpenOccupancy, leading the compared methods on those measures. Joint training helps across tasks: adding geometry and occupancy raises detection mAP by 1.2 points over detection-only training, while combining geometry with detection cuts camera-to-rig translation error by 13.4% versus geometry-only training. The predicted geometry and tracks also serve as priors for lidarless NuRec; the reported end-to-end reconstruction workflow runs 4.3× faster. The results show how one shared model can produce both structured scene understanding and inputs for simulation, while performance still depends on which tasks are trained together.