Stream3D-VLM: Online 3D Spatial Understanding with Incremental Geometry Priors

Stream3D-VLM, developed by researchers from Zhejiang University and Tencent Hunyuan, is the first online 3D vision-language model built for streaming video rather than offline clips or full-scene scans. Its core novelty is to treat “when to answer” as part of autoregressive next-token prediction, using special <SEP> and <END> tokens, so the model learns to stay silent until sufficient 3D evidence arrives. To recover geometry without explicit 3D sensor inputs, it incrementally injects temporally aligned priors from the feed-forward reconstruction model StreamVGGT through a lightweight Visual–Spatial Feature Integration module, then compresses long histories with Geometry-Adaptive Voxel Compression, which clusters 3D voxels by spatial proximity and aggregates them with dual attention. The authors also release Stream3D-1M, a 1,003,203-question dataset spanning 5,154 scans, and Stream3D-Bench, a 10,037-sample benchmark over 29 tasks with a new Answer-Timing Accuracy metric that scores both correctness and temporal precision. Based on Qwen2.5-VL-3B/7B, Stream3D-VLM-8B reaches 65.9% average accuracy on VSI-Bench, outperforming Gemini-2.5 Pro and other 72B-scale baselines, and it achieves 75.4% Answer-Timing Accuracy with only 62 ms TTFT and 0.39 s end-to-end latency on Stream3D-Bench. It also improves offline 3D tasks such as ScanQA, ScanRefer, and Scan2Cap, showing that online interaction and strong 3D spatial reasoning can coexist in one model.