Injecting Image Guidance into Text-Conditioned Diffusion Models at Inference

The paper introduces Visual Concept Fusion, a training-free inference-time method for injecting reference images into text-conditioned diffusion models such as Stable Diffusion v2 without retraining the generator or concept-specific fine-tuning. The core idea is to align CLIP image tokens to the CLIP text-token manifold with a lightweight 2-layer MLP trained on only a 10% subset of COCO Captions, then fuse aligned image and text tokens through concatenation, naive blending, or cross-attention; an optional Prompt-Noise Optimization module further refines the initial DDIM noise and conditioning tokens. Using Stable Diffusion v2.1 at 768×768 with 50 DDIM steps, the method preserves prompt semantics while transferring style, composition, and color palette from the reference image. Quantitatively, the main trade-off is between CLIP text alignment and LPIPS reference similarity: text-only SDv2 scores 0.29 CLIP and 0.78 LPIPS, naive fusion scores 0.28 and 0.77, and VCF reaches 0.27 and 0.76, indicating stronger perceptual adherence to the reference at a modest cost in prompt literalism. The aligner is lightweight, trains in under two hours on a single A100 GPU, and ablations show that combining InfoNCE with cross-attention reconstruction is necessary for balancing global semantic alignment and local visual detail; Prompt-Noise Optimization further reduces noise and improves fidelity to reference-specific features.