Scenes as Objects, Not Primitives: Instance-Structured 3D Tokenization from Unposed Views

Scenes as Objects, Not Primitives proposes a feed-forward 3D reconstruction framework from unposed multi-view images that represents a scene as instance-structured 3D token groups rather than dense per-Gaussian primitives. Built on the pretrained VGGT geometry foundation model and evaluated on ScanNet, the method uses an image-anchor transformer to decode anchor tokens that spawn 3D Gaussians, then an anchor-grouping transformer to produce instance tokens that compete for anchor ownership via softmax assignment. Training relies only on 2D supervision: RGB rendering loss for reconstruction and Hungarian-matched instance-mask supervision for grouping, with no 3D annotations. The key novelty is a two-level semantic factorization: each instance stores a shared 512-dimensional group embedding, while 8-dimensional anchor residuals capture local variation, reducing feature storage from 8.4M scalars in Uni3R to 59.4K. On ScanNet with 2 context views, the model reaches 0.661 source-view mIoU and 0.657 target-view mIoU for feature lifting, while keeping reconstruction competitive at 25.28 PSNR, 0.771 SSIM, and 0.238 LPIPS. With 8 context views, it achieves the best class-agnostic instance segmentation, reaching 0.235 AP, 0.438 AP50, and 0.564 AP25, outperforming per-scene optimization baselines such as Gaussian Grouping and ObjectGS. The same token groups also enable direct instance-level editing and open-vocabulary retrieval, and the paper reports zero-shot transfer to MipNeRF360 and robustness on RealEstate10K using SAM2 pseudo-labels.