Scalable Patch-Level Self-Supervised Learning
AuthorsMaximilian Seitzer, Gabriele Trivigno, Antonín Vobecký, Seungeun Yi, Maxime Oquab, Huy V. Vo, Oriane Siméoni, Piotr Bojanowski
AffiliationsMeta FAIR
JEM offers a principled way to learn visual representations by aligning patches across views, scaling to 7B parameters and performing strongly on dense vision tasks.
Key results
JEM trains stably at this model scale.
Tokens used to train the 7B JEM model on A140M.
DINOv3 used a dataset 12 times larger than JEM’s A140M.
JEM score, compared with DINOv3’s 27.0.
JEM score, compared with DINOv3’s 21.6.
What the paper found
Meta’s JEM is a patch-level self-supervised learning method derived from an information-theoretic objective: it aligns corresponding patch predictions across student and teacher views, explicitly prevents global representation collapse, and preserves spatial structure by matching patch-similarity patterns. Unlike DINOv2, JEM uses a unified patch-based objective rather than combining separate global and local losses, and it trains stably at 7B parameters. On A140M, its 7B model used 1.85T training tokens and, without refinement stages, achieved panoptic quality scores of 27.3 on COCONut and 23.9 on ADE20k, versus DINOv3’s 27.0 and 21.6. DINOv3 was trained on a dataset 12 times larger. JEM also consistently surpasses DINOv2 on dense prediction benchmarks while remaining competitive on image classification, suggesting principled patch-level learning can scale without a more complex collection of objectives.
Original abstract
Self-supervised learning (SSL) at scale produces powerful visual representations. However, most scalable SSL methods rely on ad hoc combinations of multiple objectives and stabilization mechanisms. Taking a step back, we ask if we can design a high-performing, yet principled SSL algorithm. Starting from the multi-view assumption, stipulating that task-relevant content is captured by the information common to different views, we construct an information-theoretic objective decomposing into interpretable terms. This derivation yields JEM, a student-teacher method that learns by aligning corresponding patch representations across views, explicitly regularized by information and structure preservation losses. JEM trains stably from 300M to 7B parameters, and, to our knowledge, is the first latent-space patch-level method demonstrated at 7B scale. Across all scales, JEM reaches strong performance on both global and dense probing tasks, on segmentation benchmarks consistently surpassing the DINOv2 algorithm, an influential foundation for today's strongest visual SSL methods. Notably, at 7B parameters, it exceeds the performance of DINOv3 on panoptic segmentation, despite being trained on $12\times$ less data without refinement stages. These results demonstrate that we can indeed design an SSL algorithm that learns strong representations, is principled and stable.
Read the original paperMore in Self-Supervised Learning
Browse all 25 papers →From Pixels, Without Pre-training: Joint Generative and Self-Supervised Representation Learning in One Model
Vicente Balmaseda, Ching-Long Lin, Tianbao Yang
SCION teaches one model to understand and generate images from scratch, combining self-supervised representation learning with high-quality image synthesis.
Less Decoder is More Encoder: Geometric Representation Learning from Novel View Synthesis
Keerthi Kaashyap, Dennis Anthony, Akshay Krishnan, Nhi Ngoc Nguyen, Jeremy Collins, James Hays, Shreyas Kousik, Animesh Garg
SNAP shows that making the decoder simpler can help an encoder learn more transferable 3D-aware visual representations from self-supervised novel view synthesis.
Self-Play Pretraining with Zero Data
Aditya Cowsik, Kfir Dolev, Michael Y. Li, G. Bruno De Luca, Nourya Cohen, Noah D. Goodman, Yoav Levine
A learner and an RL-powered program generator teach each other from scratch, producing synthetic data that enables surprisingly meaningful transfer to natural datasets.