Less Decoder is More Encoder: Geometric Representation Learning from Novel View Synthesis
AuthorsKeerthi Kaashyap, Dennis Anthony, Akshay Krishnan, Nhi Ngoc Nguyen, Jeremy Collins, James Hays, Shreyas Kousik, Animesh Garg
AffiliationsGeorgia Institute of Technology
SNAP shows that making the decoder simpler can help an encoder learn more transferable 3D-aware visual representations from self-supervised novel view synthesis.
Key results
Approximate combined scale of RealEstate10K, DL3DV, and Co3Dv2 training data.
SNAP point-correspondence precision; VGGT scores 12.85%.
SNAP model parameter count, including DINOv3.
GB of peak GPU memory on an NVIDIA RTX PRO 6000.
SNAP inference speedup.
What the paper found
SNAP asks why novel-view synthesis often produces weak geometric features despite requiring models to reason about 3D. Its answer is architectural: powerful decoders can do spatial reasoning themselves, while RGB reconstruction rewards image-specific detail. SNAP instead uses a pose-free multi-view transformer encoder and a pose-conditioned decoder whose BlockNN mask restricts spatial attention to a single patch, forcing scene geometry into the encoder. It predicts frozen DINOv3 features rather than pixels, with a gradient loss to preserve local structure. Trained without dense 3D labels on RealEstate10K, DL3DV, and Co3Dv2—about 117K sequences—SNAP transfers across correspondence, localization, pose, depth, and robot manipulation. On RealEstate10K, it reaches 15.48% point-correspondence precision at 5 pixels, compared with 12.85% for geometry-supervised VGGT. Its features also remain more robust than standard 2D representations under camera shifts on MimicGen. In inference tests on an NVIDIA RTX PRO 6000, SNAP uses 233.38M parameters and, at 100 views, needs 1.03 GB of memory while running 2.9 times faster than VGGT. The central result is that better transferable geometry can come from constraining the decoder and improving the prediction target, rather than maximizing view-reconstruction capacity.
Original abstract
This paper examines the role of Novel View Synthesis (NVS) in geometric representation learning. In principle, NVS should reason about 3D scene structure, thereby enabling transferable multi-view geometric representations. Yet, existing encoder-based NVS methods yield poor representations. This is not because of a lack of supervisory signal, but rather due to inconspicuous architectural choices: \textit{spatially expressive decoders} that dilute representational capabilities of the scene encoder, and \textit{low-level pixel-space targets} that hinder feature learning. We present SNAP, a self-supervised encoder-decoder transformer that addresses both through a pose-conditioned local decoder and a latent-space reconstruction objective. SNAP is task agnostic, and we show that it is competitive with special-purpose geometry-supervised methods. SNAP also performs competitively against self-supervised representations across five tasks: visual localization, pose estimation, point correspondence, depth estimation, and robot manipulation. Remarkably, SNAP's patch features exhibit emergent viewpoint invariance that approaches heavily supervised models despite lower compute and data budgets. Under camera shifts where standard 2D representations collapse, SNAP degrades more gracefully, revealing that restricting decoder expressivity actively prevents the suppression of transferable geometric structure. https://snap-nvs.github.io
Read the original paperMore in Self-Supervised Learning
Browse all 25 papers →From Pixels, Without Pre-training: Joint Generative and Self-Supervised Representation Learning in One Model
Vicente Balmaseda, Ching-Long Lin, Tianbao Yang
SCION teaches one model to understand and generate images from scratch, combining self-supervised representation learning with high-quality image synthesis.
Scalable Patch-Level Self-Supervised Learning
Maximilian Seitzer, Gabriele Trivigno, Antonín Vobecký, Seungeun Yi, Maxime Oquab, Huy V. Vo, Oriane Siméoni, Piotr Bojanowski
JEM offers a principled way to learn visual representations by aligning patches across views, scaling to 7B parameters and performing strongly on dense vision tasks.
Self-Play Pretraining with Zero Data
Aditya Cowsik, Kfir Dolev, Michael Y. Li, G. Bruno De Luca, Nourya Cohen, Noah D. Goodman, Yoav Levine
A learner and an RL-powered program generator teach each other from scratch, producing synthetic data that enables surprisingly meaningful transfer to natural datasets.