NTH

Probing Spatial State Representations in Visual Foundation Models

AuthorsBaiqiao Yin, Zhongang Cai, Ruisi Wang, Qiwei Liang, Lei Yang, Ziwei Liu

Affiliations[

November 8, 2026 2 min read
Watch on YouTube
The one-line take

The study tests whether visual foundation models can track and use spatial information beyond what is currently in view.

Key results

32.79
Fast3R object-persistence angular error

Lowest out-of-view localization error, in degrees.

0.76
Persistence-camera-pose correlation

Spearman rank correlation across the ten representations.

0.2373
DINOv2 direct-prediction R@1

Global retrieval score for future-feature prediction.

0.1588
V-JEPA2 path-integration R@1

Global retrieval score for sequential motion integration.

0.2459
V-JEPA2 predicted-feature depth MAE

Normalized-depth MAE using the frozen reader on native predictions at horizon two.

What the paper found

This paper tests whether frozen visual foundation models maintain a useful spatial state, rather than merely encoding what is visible, across spatial structure, object persistence, predictive spatial update, and goal-directed action. Frozen-feature probes across ten representations, including DINOv2, V-JEPA2, VideoMAE, Fast3R, Wan, CogVideoX, and Alibaba’s Qwen2.5-VL, reveal distinct capability profiles: Fast3R gives the lowest out-of-view localization error at 32.79°, while DINOv2 leads direct future-feature retrieval with R@1 of 0.2373 and V-JEPA2 leads sequential path integration at 0.1588. Across models, object persistence tracks camera-pose readout, with a Spearman correlation of ρ = 0.76, rather than depth or instance recognition. The probes also expose limits: longer visual prefixes worsen object persistence across all eight models tested, and a reader trained on real V-JEPA2 features transfers imperfectly to native predictions, where its depth MAE is 0.2459. Goal-conditioned actions respond to changes in the goal, but the correct goal does not reliably improve progress. The central finding is that spatial information being decodable does not guarantee that a model maintains it, updates it accurately, or uses it effectively for action.

Original abstract

An observer sees only part of the surrounding world, yet must keep track of what lies beyond the current view. Spatial perception therefore involves more than recognizing individual images: it calls for a persistent spatial state that is updated under self-motion and used to guide action. Do visual foundation models provide representations that support such a state? We study this question through four levels of spatial capability---spatial structure, object persistence, predictive spatial update, and actionability---using frozen-feature probes on ten representations under streaming egocentric observations of static indoor environments. The resulting capability profiles vary across tasks, network layers, and prefix lengths: strong readout of the visible scene does not ensure accurate object persistence or future-feature retrieval. Matched controls separate readout accuracy from reliance on visual history and motion. Native future prediction retains recoverable spatial structure, but a spatial reader trained on real features does not transfer to predicted features unchanged. Finally, goal-dependent candidate selection does not establish reliable continuous action utility: predicted actions respond to goal changes, yet supplying the correct goal does not reliably improve progress. Across the four levels, object persistence and actionability build on camera-pose readout, whereas feature prediction builds on visible-scene readout. Together, these findings motivate evaluating spatial representations not only by what can be decoded from them, but by whether that information is maintained beyond observation, updated predictively, and used for goal-directed action.

Read the original paper

More in Foundation Models

Browse all 53 papers →
01Foundation Model

Towards Foundation Decision Models: Drex

Nischay Dhankhar, Alisher Amantay, Zhanibek Datbayev, Darkhan Akhmetov, Yash Shah, Nurmukhamed Ubaidullayev, Raman Saparkhan, Dos Baha, Abulhair Saparov

Drex aims to make AI decisions faster and more consistent by predicting directly among options instead of generating long answers.

Read analysis
02Foundation Model

Venus-DefakerOne2: A Vision-Forensics Foundation Model with Broader Coverage

Chenfan Qu, Yachao Liang, Mingqi Fang, Jiangwei Xie, Zhenming Wang, Baoyue Zhang, Changjiang Jiang, Zheng Chang, Fanli Jin, Xuekang Zhu, Kaiwen Feng, Junchi Li, Weihong Zhang, Haoyue Wang, Chenfeng Zhang, Yinan Xia, Fangming Cui, Dong Fan, Jingjing Liu, Tianyi Zhang, Luobin Chen, Ji-Zhe Zhou, Sheng Li, Wei Wang, Bin Li, Ran He, Xinpeng Zhang, Linna Zhou, Jian Liu, Weiqiang Wang

Venus-DefakerOne2 aims to detect many kinds of image and video forgery with one broadly trained vision-forensics model.

Read analysis
03Foundation Model

Nemotron 3.5 Super and Lightning: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Models for Agentic AI

NVIDIA, :, Aastha Jhunjhunwala, Aayushi Ashokkumar Srivastava, Abhibha Gupta, Abhinav Garg, Abhinav Khattar, Achsah Wildish, Adam Rajfer, Adi Renduchintala, Adil Asif, Adriana Flores Miranda, Aileen Zaman, Aishwarya Chhabra, Akanksha Shukla, Akhiad Bercovich, Alejandra Rico, Aleksa Gordic, Aleksander Ficek, Alex Gronskiy, Alex Steiner, Alexander Bukharin, Alexandre Milesi, Ali Roshan Ghias, Ali Shehper, Ali Taghibakhshi, Alice Gatti, Alisa Liu, Alok Kumar, Alok Tripathy, Amala Sanjay Deshmukh, Aman Miryala, Aman Shanbhag, Ameya Sunil Mahabaleshwarkar, Ameya Velingker, Amir Klein, Amir Samani, Amit Zuker, Amnon Geifman, Anahita Bhiwandiwalla, Ananth Subramaniam, Andre Manoel, Andrea Santilli, Andrew Tao, Andrii Skliar, Anil Thomas, Anish Mahishi, Anna Shors, Anna Warno, Anuj Doshi, Anusha Kamath, Anwith Kiran, Ao Tang, Arham Mehta, Arnab Dhara, Arnav Komaragiri, Arnav Kundu, Arti Jain, Arun Rao, Arun Venkatesan, Arushi Goel, Asha Anoosheh, Ashish Vaswani, Ashwath Aithal, Asif Ahame...

Nemotron 3.5 pairs hybrid Mamba-Transformer models with agent training and inference optimizations to deliver capable, efficient open AI systems.

Read analysis