FrameForge: Harnessing Priors of Video Foundation Models for Unified Image Generation
AuthorsYijing Lin, Zhendong Mao, Mengqi Huang, Yuxin Wang, Yongdong Zhang
AffiliationsIntroduction With recent advances in generative modeling [8, 19, 21, 33, 35, 57, 67], foundation models such as Stable Diffusion [13, 14, 41] and FLUX [25] offer strong text‑to‑image generation ca
FrameForge uses video models’ sense of temporal change to better generate images that preserve a visual input while following new instructions.
Key results
Total examples used across the three generation tasks.
Parameters in the shared LoRA adaptation.
Overall score for subject-driven generation.
Structural similarity score for canny-to-image generation.
Overall score for instruction-guided image editing.
GEdit-Bench score for motion-change editing.
What the paper found
FrameForge adapts the HunyuanVideo1.5 video-generation model for one shared image generator spanning subject-driven creation, canny-guided synthesis, and instruction-based editing. Its Mixture of Temporal Experts selects among virtual temporal-position embeddings according to each input and denoising step, while Chain of Temporal Reasoning adds a latent intermediate pathway that encourages the model to represent how a condition becomes a target. The backbone stays frozen, and a shared LoRA is trained on 510K examples. On DreamBench, FrameForge scores 0.739 overall; on COCO2017, its SSIM is 0.447, a 7.2% improvement over the next-best method. On GEdit-Bench, it reaches 6.12 overall and 6.84 for motion changes, exceeding the open-source comparison while its motion-change score remains below GPT-4o but above Gemini 2.5 and Seedream 4.0. The model has 182M trainable parameters, and ablations show that combining adaptive temporal selection with the reasoning pathway improves results across the three tasks.
Original abstract
Unified conditional image generation requires a single model to preserve different aspects of a visual input while producing a target image that follows textual guidance. Pretrained video models provide a promising foundation by encoding visual transformations across time, yet existing approaches typically assign the condition and target to fixed temporal positions. We observe that condition-target relationships vary substantially across generation tasks, while video models naturally encode visual transformations at different temporal scales. This motivates us to model conditional generation through input-adaptive virtual temporal relations. We propose FrameForge, which captures two complementary aspects of such relations. Mixture of Temporal Experts (MoTE) models how far the target lies from the condition by dynamically selecting temporal RoPE experts with different virtual intervals. Chain of Temporal Reasoning (CoTR) models how the condition is transformed into the target through a latent intermediate state, with a reasoning-bottleneck attention mask encouraging information propagation through this pathway. Experiments on DreamBench, COCO2017, and GEdit-Bench show that FrameForge achieves superior performance in subject-driven generation, canny-to-image generation, and image editing among the compared methods.
Read the original paperMore in Generative Models
Browse all 87 papers →dNaViT: Discrete Codes are General-Purpose Visual Representations
Xiaotong Li, Zhengrong Yue, Xu Huang, Jialv Zou, Minhao Jing, Wenhao Yan, Haodong Li, Hai Huang, Bo Zhao, Yan Feng, Peng Pei, Xunliang Cai
dNaViT turns images into reusable discrete codes designed to work across visual understanding, generation, and editing.
Lipschitz-Controlled Flow Matching for Robust Image Restoration
Sedjro Salomon Hotegni, Konstantin Sonntag, Paul-Christian Bürkner, Sebastian Peitz
LipsFlower aims to make flow-matching image restorations less sensitive to small input changes without sacrificing perceptual quality.
PhysLoop: Harnessing LLMs' Physics Awareness for Self-Improving Video Generation Models
Yuxuan Chen, Xiaojie Li, Hao Chen, Juncheng Ma, Kewei Zhang, Zhongjing Du, Guoqiang Wei, Fangfang Wang, Zhen Dong, Zhiyang Dou, Liming Zhao, Yan Zeng, Jianwei Zhang, Daquan Zhou
PhysLoop uses LLMs’ physics knowledge to help video generators create and repair more physically consistent videos.