Text Template Tokens Are Implicit Semantic Registers in Diffusion Transformers

Researchers from Nanjing University, Alibaba Group, and Zhejiang University analyze how chat-formatted text controls diffusion transformers, focusing on Qwen-Image and Qwen-Image-2512. Their causal interpretability framework combines attention decomposition, cross-prompt swaps, head transplantation, and layer-wise masking. The central finding is that trailing structural tokens—such as dialogue delimiters—contain little prompt-specific meaning at the vision-language encoder output, yet become dominant image-to-text attention sinks inside the DiT and function as implicit semantic registers. On GenEval, these template tokens receive 76% of image-to-text attention, and each structural token attracts 6.4 times as much attention as an average prompt token. Causal interventions show that semantics are not transferred directly from prompt tokens into the registers: prompt tokens first inject object identity into image latents, and the structural tokens then read that identity back from the image stream. Transplanting only about 18% of the right attention heads can change an image from an apple to a banana, while heads that read prompt tokens most strongly are largely dispensable. The authors convert this result into a training-free pruning method over 1440 heads: pruning selected heads during late denoising removes 20% of attention FLOPs, reducing GenEval accuracy from 76.1 to 74.7, a 1.4-point drop. The study further separates semantic routing from visual rendering and identifies a depth-wise sequence of early identity commitment, middle-layer propagation, and late refinement.