From Pixels, Without Pre-training: Joint Generative and Self-Supervised Representation Learning in One Model
AuthorsVicente Balmaseda, Ching-Long Lin, Tianbao Yang
AffiliationsTexas A&M University · University of Iowa
SCION teaches one model to understand and generate images from scratch, combining self-supervised representation learning with high-quality image synthesis.
Key results
Unconditional ImageNet 256×256 generation.
JiT with Dispersive Loss, compared with SCION's 8.92.
SCION's guided generation result, versus 5.01 for DINOv2-dependent RCG.
Linear-probe accuracy of SCION's learned representation.
Average transfer accuracy across eight datasets.
Unconditional ImageNet generation with SCION at larger scale.
What the paper found
SCION combines image generation and visual representation learning in one pixel-space model trained from scratch, without labels, pretrained encoders, autoencoders, or tokenizers. Its shared JiT encoder uses clean augmented images to learn semantic [CLS] embeddings through contrastive learning, then uses those embeddings to condition flow matching on noisy images; patch tokens retain the spatial detail needed for synthesis. A learned embedding prior supplies conditioning vectors at generation time, while stop-gradient operations and gradient-norm balancing help reconcile the objectives. On ImageNet at 256×256, JiT-B/16 SCION reaches 8.92 FID, compared with 54.14 for the strongest self-contained baseline, and representation guidance lowers FID to 4.20, beating DINOv2-dependent RCG at 5.01. Its learned features also reach 72.35% ImageNet top-1 accuracy and 79.89% average transfer accuracy across eight datasets. At larger scale, JiT-L/16 with representation guidance achieves 3.47 FID. Unlike systems that import semantics from DINOv2, SCION learns its representations and image generator jointly; the reported experiments used NVIDIA H200 GPUs.
Original abstract
Strong image generation models are conditioned on class labels, aligned to frozen pretrained encoders, or built on separately trained autoencoders. While effective, generation then depends on supervision or pretraining: labels must be annotated, and encoders or autoencoders pretrained for the target domain. We study joint generative and self-supervised representation learning in a single model, enabling self-conditioned generation without labels or pretrained models. This is challenging because the objectives are mismatched: contrastive learning consumes clean augmented views and favors coarse, invariant semantics, while flow matching consumes noisy images and must preserve the fine detail and spatial layout that contrastive learning discards. We propose SCION (Self-conditioned Generation on Self-supervised representation), whose core is a single pixel-space encoder conditioned on the flow timestep and an embedding. For representation learning, this conditioning embedding is a learned global vector shared across images, with the encoder's [CLS] token yielding the semantic representation trained by the contrastive loss. For generative training, the conditioning embedding is the image's own [CLS] representation, while patch tokens pass through a decoder to predict the image. To sample without a reference image at inference, we jointly learn a prior over the embedding. Gradient-norm balancing and stop-gradient mechanisms enable joint optimization in one run. SCION is self-supervised and self-contained, with no labels or pretrained models. On ImageNet 256x256, with the JiT-B recipe and no representation guidance, SCION reaches 8.92 FID, surpassing class-unconditional iREPA, which aligns to pretrained DINOv2 (46.44), and RCG, which conditions on it (14.27). With JiT-L, SCION achieves 5.89 FID without guidance and 3.47 with representation guidance, outperforming RCG with the ADM recipe (6.24).
Read the original paperMore in Self-Supervised Learning
Browse all 25 papers →Less Decoder is More Encoder: Geometric Representation Learning from Novel View Synthesis
Keerthi Kaashyap, Dennis Anthony, Akshay Krishnan, Nhi Ngoc Nguyen, Jeremy Collins, James Hays, Shreyas Kousik, Animesh Garg
SNAP shows that making the decoder simpler can help an encoder learn more transferable 3D-aware visual representations from self-supervised novel view synthesis.
Scalable Patch-Level Self-Supervised Learning
Maximilian Seitzer, Gabriele Trivigno, Antonín Vobecký, Seungeun Yi, Maxime Oquab, Huy V. Vo, Oriane Siméoni, Piotr Bojanowski
JEM offers a principled way to learn visual representations by aligning patches across views, scaling to 7B parameters and performing strongly on dense vision tasks.
Self-Play Pretraining with Zero Data
Aditya Cowsik, Kfir Dolev, Michael Y. Li, G. Bruno De Luca, Nourya Cohen, Noah D. Goodman, Yoav Levine
A learner and an RL-powered program generator teach each other from scratch, producing synthetic data that enables surprisingly meaningful transfer to natural datasets.