Scal3R: Learning Efficient Multi-Relative Pose Query for Scalable Online 3D Reconstruction

Scal3R addresses the long-video collapse seen in online 3D reconstruction systems such as CUT3R and STream3R, which regress every camera pose against a distant first-frame anchor. It observes that local depth remains stable while global pose drifts, then replaces absolute regression with multi-reference relative pose querying. Lightweight learnable pose tokens, adding only about 1% of backbone parameters, are injected into a completely frozen backbone through asymmetric attention: pose tokens read image features, while image tokens remain unchanged for point-cloud generation. Each incoming frame queries multiple keyframes, and online Pose-Graph Optimization with iSAM2, keyframe selection, and DINOv2-SALAD loop closure integrates the relative constraints into a globally consistent trajectory. Trained on TartanAir using 4-view samples, Scal3R converges in 8 hours on a single NVIDIA A100 GPU and uses K=12 references at inference. On KITTI, it reaches an average ATE of 69.7 versus 182.2 for TTT3R, reducing error by over 60%; on Virtual KITTI, the CUT3R variant achieves 5.63 ATE. The method also reports state-of-the-art results across Sintel, TUM-Dynamic, ScanNet, and 7-Scenes, while preserving real-time operation at 14.4 FPS with loop closure. Its central contribution is shifting long-range reconstruction from fragile global extrapolation to locally reliable relative constraints aggregated by an online graph.