Text-to-Image Generative Modeling Without Language Pretraining
AuthorsKeshigeyan Chandrasegaran, Yue Zhao, Armin W. Thomas, Ehsan Adeli, Jiajun Wu, Juan Carlos Niebles, Li Fei-Fei, Michael Poli
AffiliationsStanford University · Radical Numerics
Halcyon shows that capable text-to-image models can be trained from scratch on image-text pairs, without first pretraining a language model.
Key results
Dataset used to train Halcyon from scratch.
Compared with 63.3 for LlamaGen-XL/8.
Halcyon-0.52B uses this many times less total pretraining compute.
Compared with 41.0 for JiT-T2I-XXL/8++.
Halcyon-1.6B uses this many times less total pretraining compute.
What the paper found
Halcyon tests whether text-to-image generation needs internet-scale language pretraining. Instead of relying on a separate pretrained text encoder, it trains a decoder-only Transformer from scratch on GPIC, a dataset of 100M image-text pairs, using a shared text-image vocabulary and next-token prediction. On GPIC, Halcyon-0.52B reaches 48.0 gFD versus LlamaGen-XL/8’s 63.3, with 18.2× less total pretraining compute; Halcyon-1.6B reaches 35.3 versus JiT-T2I-XXL/8++ at 41.0, using 275× less compute. Controlled experiments find that text-only pretraining—including initialization from Qwen3—does not improve generation under the tested settings. The team also shows that replacing caption dropout with image-to-text supervision enables captioning while keeping text-to-image quality competitive under the same training budget. The central result is that paired image-text data alone can teach effective text conditioning, challenging the assumption that strong image generation requires a pretrained language model.
Original abstract
Modern text-to-image (T2I) systems use internet-scale language pretraining for text-conditioned image generation. We ask whether language pretraining is necessary for strong T2I generation. We introduce Halcyon, a family of early-fusion multimodal autoregressive models pretrained entirely from scratch on GPIC, a dataset of 100M image-text pairs. Halcyon-0.52B outperforms LlamaGen-XL/8 (48.0 vs. 63.3 gFD) with 18.2$\times$ less pretraining compute. Further, Halcyon-1.6B outperforms JiT-T2I-XXL/8++ (35.3 vs. 41.0 gFD) with 275$\times$ less pretraining compute. Controlled experiments show that text-only pretraining does not improve T2I generation. Halcyon also learns image captioning by replacing caption dropout with I2T supervision, maintaining competitive T2I generation quality under the same pretraining budget. We show that strong T2I systems can be trained from scratch without internet-scale language pretraining. Code and models are available at https://cs.stanford.edu/~keshik/halcyon
Read the original paperMore in Generative Models
Browse all 78 papers →Qwen-Image-PE: Learning Visual Design for Text-to-Image Prompt Enhancement
Zekai Zhang, Yanran Zhang, Shengming Yin, Yan Shu, Zhening Liu, Bingxuan Zhao, Chenyang Zhu, Donghao Zhou, Jiahao Li, Jiahao Wang, Jie Zhang, Kaiyuan Gao, Kun Yan, Lihan Jiang, Ningyuan Tang, Qihang Ai, Rui Song, Shuo Cao, Xiao Xu, Xiaoyue Chen, Yuxiang Chen, Zhendong Wang, Zhouyang Li, Zihao Liu, Zikai Zhou, Bo Zheng, Huishuai Zhang, Dongyan Zhao, Chenfei Wu
Qwen-Image-PE turns brief image requests into more carefully designed prompts to improve the images generated by text-to-image models.
T-DIG: Text-Conditioned Diversity for Image Generation
Simran Khanuja, Rohit Gandikota, Gaurav Parmar, Graham Neubig
T-DIG lets users steer how a text-to-image system varies its results, without needing to modify the underlying image generator.
InfiniDream: Open-Ended Photorealistic 3D Worlds Through Procedural Generation
Yi-Chuan Huang, Hao-Jen Chien, Zheng-Hui Huang, Yu-Lun Liu, Kaipeng Zhang, Zhixiang Wang
InfiniDream uses language-guided procedural tools to build editable, photorealistic 3D assets and populate expansive worlds.