Probing Spatial State Representations in Visual Foundation Models
AuthorsBaiqiao Yin, Zhongang Cai, Ruisi Wang, Qiwei Liang, Lei Yang, Ziwei Liu
Affiliations[
The study tests whether visual foundation models can track and use spatial information beyond what is currently in view.
Key results
Lowest out-of-view localization error, in degrees.
Spearman rank correlation across the ten representations.
Global retrieval score for future-feature prediction.
Global retrieval score for sequential motion integration.
Normalized-depth MAE using the frozen reader on native predictions at horizon two.
What the paper found
This paper tests whether frozen visual foundation models maintain a useful spatial state, rather than merely encoding what is visible, across spatial structure, object persistence, predictive spatial update, and goal-directed action. Frozen-feature probes across ten representations, including DINOv2, V-JEPA2, VideoMAE, Fast3R, Wan, CogVideoX, and Alibaba’s Qwen2.5-VL, reveal distinct capability profiles: Fast3R gives the lowest out-of-view localization error at 32.79°, while DINOv2 leads direct future-feature retrieval with R@1 of 0.2373 and V-JEPA2 leads sequential path integration at 0.1588. Across models, object persistence tracks camera-pose readout, with a Spearman correlation of ρ = 0.76, rather than depth or instance recognition. The probes also expose limits: longer visual prefixes worsen object persistence across all eight models tested, and a reader trained on real V-JEPA2 features transfers imperfectly to native predictions, where its depth MAE is 0.2459. Goal-conditioned actions respond to changes in the goal, but the correct goal does not reliably improve progress. The central finding is that spatial information being decodable does not guarantee that a model maintains it, updates it accurately, or uses it effectively for action.
Original abstract
An observer sees only part of the surrounding world, yet must keep track of what lies beyond the current view. Spatial perception therefore involves more than recognizing individual images: it calls for a persistent spatial state that is updated under self-motion and used to guide action. Do visual foundation models provide representations that support such a state? We study this question through four levels of spatial capability---spatial structure, object persistence, predictive spatial update, and actionability---using frozen-feature probes on ten representations under streaming egocentric observations of static indoor environments. The resulting capability profiles vary across tasks, network layers, and prefix lengths: strong readout of the visible scene does not ensure accurate object persistence or future-feature retrieval. Matched controls separate readout accuracy from reliance on visual history and motion. Native future prediction retains recoverable spatial structure, but a spatial reader trained on real features does not transfer to predicted features unchanged. Finally, goal-dependent candidate selection does not establish reliable continuous action utility: predicted actions respond to goal changes, yet supplying the correct goal does not reliably improve progress. Across the four levels, object persistence and actionability build on camera-pose readout, whereas feature prediction builds on visible-scene readout. Together, these findings motivate evaluating spatial representations not only by what can be decoded from them, but by whether that information is maintained beyond observation, updated predictively, and used for goal-directed action.
Read the original paperMore in Foundation Models
Browse all 53 papers →Towards Foundation Decision Models: Drex
Nischay Dhankhar, Alisher Amantay, Zhanibek Datbayev, Darkhan Akhmetov, Yash Shah, Nurmukhamed Ubaidullayev, Raman Saparkhan, Dos Baha, Abulhair Saparov
Drex aims to make AI decisions faster and more consistent by predicting directly among options instead of generating long answers.
Venus-DefakerOne2: A Vision-Forensics Foundation Model with Broader Coverage
Chenfan Qu, Yachao Liang, Mingqi Fang, Jiangwei Xie, Zhenming Wang, Baoyue Zhang, Changjiang Jiang, Zheng Chang, Fanli Jin, Xuekang Zhu, Kaiwen Feng, Junchi Li, Weihong Zhang, Haoyue Wang, Chenfeng Zhang, Yinan Xia, Fangming Cui, Dong Fan, Jingjing Liu, Tianyi Zhang, Luobin Chen, Ji-Zhe Zhou, Sheng Li, Wei Wang, Bin Li, Ran He, Xinpeng Zhang, Linna Zhou, Jian Liu, Weiqiang Wang
Venus-DefakerOne2 aims to detect many kinds of image and video forgery with one broadly trained vision-forensics model.
Nemotron 3.5 Super and Lightning: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Models for Agentic AI
NVIDIA, :, Aastha Jhunjhunwala, Aayushi Ashokkumar Srivastava, Abhibha Gupta, Abhinav Garg, Abhinav Khattar, Achsah Wildish, Adam Rajfer, Adi Renduchintala, Adil Asif, Adriana Flores Miranda, Aileen Zaman, Aishwarya Chhabra, Akanksha Shukla, Akhiad Bercovich, Alejandra Rico, Aleksa Gordic, Aleksander Ficek, Alex Gronskiy, Alex Steiner, Alexander Bukharin, Alexandre Milesi, Ali Roshan Ghias, Ali Shehper, Ali Taghibakhshi, Alice Gatti, Alisa Liu, Alok Kumar, Alok Tripathy, Amala Sanjay Deshmukh, Aman Miryala, Aman Shanbhag, Ameya Sunil Mahabaleshwarkar, Ameya Velingker, Amir Klein, Amir Samani, Amit Zuker, Amnon Geifman, Anahita Bhiwandiwalla, Ananth Subramaniam, Andre Manoel, Andrea Santilli, Andrew Tao, Andrii Skliar, Anil Thomas, Anish Mahishi, Anna Shors, Anna Warno, Anuj Doshi, Anusha Kamath, Anwith Kiran, Ao Tang, Arham Mehta, Arnab Dhara, Arnav Komaragiri, Arnav Kundu, Arti Jain, Arun Rao, Arun Venkatesan, Arushi Goel, Asha Anoosheh, Ashish Vaswani, Ashwath Aithal, Asif Ahame...
Nemotron 3.5 pairs hybrid Mamba-Transformer models with agent training and inference optimizations to deliver capable, efficient open AI systems.