What, Where, and How: Probing Spatiotemporal Representations in Video Foundation Models

This study examines what, where, and how temporal information is encoded in the self-supervised video foundation models V-JEPA 2, developed by Meta, and VideoMAE-v2. Using frozen representations and lightweight linear SVM probes across transformer depth, the researchers test CameraBench, IntPhys 2, and UBnormal for camera motion, intuitive physics, and anomaly detection. Camera motion is strongly represented: VideoMAE-v2 Giant reaches 93.8% mean test ROC-AUC, while V-JEPA 2 Large and Giant reach 91.0% and 90.2%, showing that scaling does not guarantee better generalization. Intuitive-physics performance remains near 50% ROC-AUC across layers, indicating little linearly accessible knowledge of solidity, permanence, continuity, or immutability in the more challenging IntPhys 2 benchmark. Anomaly detection is intermediate, with V-JEPA 2 Giant reaching 68.4% test ROC-AUC versus 62.1% for VideoMAE-v2 Giant, and its strongest signals emerging in deeper layers. Beyond classification, PCA of per-temporal-patch features from RealEstate10K reveals smooth, curved, low-dimensional trajectories that track camera motion, although some clips exhibit oscillatory geometry. The paper then introduces geometry-aware cubic-spline, or tangent-based, steering, which follows these empirical trajectories instead of cutting through latent space with linear interpolation. Nearest-neighbor retrieval across 200 evaluation clips shows spline steering consistently stays closer to the original representation manifold and produces smoother, more temporally coherent frame progressions, suggesting that video latent spaces are not only decodable but also geometrically navigable.