Lipschitz-Controlled Flow Matching for Robust Image Restoration
AuthorsSedjro Salomon Hotegni, Konstantin Sonntag, Paul-Christian Bürkner, Sebastian Peitz
AffiliationsPaderborn University · TU Dortmund University
LipsFlower aims to make flow-matching image restorations less sensitive to small input changes without sacrificing perceptual quality.
Key results
Shared test set for blind face restoration.
Flow-from-Y under RMS-constrained SPSA; maximum gain falls from 1.74 to 1.16.
Flow-from-Y under RMS-constrained SPSA; worst-five-percent average gain falls from 0.849 to 0.747.
Flow-from-Y under RMS-constrained SPSA.
Flow-from-Y restoration quality; PSNR changes from 25.784 to 25.475 dB.
What the paper found
LipsFlower targets a weakness in generative image restoration: small changes to a degraded image can produce large changes in the reconstruction. Its stability analysis distinguishes two pathways—perturbations entering through the initial state in Flow-from-Y, and perturbations affecting the velocity field throughout observation-conditioned sampling. To limit amplification, it introduces LipHDiT, an hourglass diffusion transformer with stabilized approximate spectral normalization, bounded modulation, and convex residual mixing, alongside adversarial finite-radius gain penalties at sampled flow states. On 3000 CelebA-Test face pairs at 512×512 resolution, the Flow-from-Y version reduces worst-case gain under an RMS-constrained SPSA attack from 1.74 to 1.16, a 33% decrease; its worst-five-percent average gain falls from 0.849 to 0.747, about 12%, and LPIPS drift drops about 26%. The robustness gain comes with a 0.31 dB PSNR decrease, from 25.784 to 25.475 dB. The conditional BayesFlow version lowers maximum amplification and perceptual drift but sacrifices more restoration quality. Comparisons include CodeFormer and OSDFace, which uses Stable Diffusion 2.1-base. The controls encourage stability but do not provide a global Lipschitz guarantee, and the evaluation is limited to blind face restoration.
Original abstract
Flow-matching models can produce high-quality image restorations, but their learned dynamics may amplify small input perturbations and make reconstructions unstable. We introduce LipsFlower to reduce this amplification. We derive stability bounds for Flow-from-$Y$, which initializes the ordinary differential equation from a noise-perturbed observation, as well as for observation-conditioned posterior sampling, which conditions the velocity field on the observation throughout integration. Based on this analysis, we develop LipHDiT, a Lipschitz-aware hourglass diffusion transformer that serves as the vector-field architecture of LipsFlower. LipHDiT incorporates stabilized approximate spectral normalization, bounded modulation, and convex residual mixing. We also use adversarial finite-radius gain regularization at sampled interpolation states to penalize excessive sensitivity to the state and, for conditional sampling, the observation. We test both formulations on blind face restoration, measuring adversarial robustness and the perceptual quality of the reconstructions. LipsFlower reduces perturbation amplification while keeping restoration quality competitive with the evaluated baselines. These results suggest that Lipschitz-aware flow matching can improve the stability of generative image restoration.
Read the original paperMore in Generative Models
Browse all 87 papers →dNaViT: Discrete Codes are General-Purpose Visual Representations
Xiaotong Li, Zhengrong Yue, Xu Huang, Jialv Zou, Minhao Jing, Wenhao Yan, Haodong Li, Hai Huang, Bo Zhao, Yan Feng, Peng Pei, Xunliang Cai
dNaViT turns images into reusable discrete codes designed to work across visual understanding, generation, and editing.
PhysLoop: Harnessing LLMs' Physics Awareness for Self-Improving Video Generation Models
Yuxuan Chen, Xiaojie Li, Hao Chen, Juncheng Ma, Kewei Zhang, Zhongjing Du, Guoqiang Wei, Fangfang Wang, Zhen Dong, Zhiyang Dou, Liming Zhao, Yan Zeng, Jianwei Zhang, Daquan Zhou
PhysLoop uses LLMs’ physics knowledge to help video generators create and repair more physically consistent videos.
FrameForge: Harnessing Priors of Video Foundation Models for Unified Image Generation
Yijing Lin, Zhendong Mao, Mengqi Huang, Yuxin Wang, Yongdong Zhang
FrameForge uses video models’ sense of temporal change to better generate images that preserve a visual input while following new instructions.