Appearance Pointers -- Multimodal Region Control of Diffusion Transformers

Researchers from Brown University and Adobe Research introduce Appearance Pointers, a modality-agnostic control interface for Diffusion Transformers, demonstrated with Black Forest Labs’ FLUX model. The method aligns text descriptions, reference images, and user-defined region masks through a Region Correspondence Transformer, then compresses multiple regional signals into spatially aggregated pointer tokens. These pointers direct the DiT toward the correct appearance cues and locations during a single denoising pass, while region-contour guidance improves boundary precision. The same model supports fine and sparse layout generation, multi-subject insertion, pose control, editing, and simultaneous image-plus-text conditioning without retraining the base model from scratch. The authors create the synthetic AppearancePointers-37K dataset and evaluate on a 500-image benchmark averaging five regions per image. For text-conditioned regions, the method reaches DINO-I 56.09 and CLIP-IQA 95.02; for image-conditioned regions, it achieves DINO-I 69.31, CLIP-I 93.29, and MIoU 40.97, surpassing MSDiffusion and DreamRenderer on identity and spatial adherence. The added correspondence and aggregation modules contain approximately 400M parameters, a 3.33% increase over the base model, while pointer computation occurs once per generation rather than at every diffusion step. Ablations show that removing region aggregation reduces DINO-I to 54.47, and removing contour guidance reduces MIoU to 35.81. Limitations include weaker fine-grained face identity preservation and degradation when conditioning on about 10 or more regions.