Training Vision Transformers to Focus: Emergence of Spatial Attention Head Specialization
AuthorsYuji Nozawa, Yu-Chieh Lin, Youyang Ng
AffiliationsKioxia Corporation
The method encourages Vision Transformers to develop distinct, spatially focused attention heads without image annotations.
Key results
Combined losses; DINOv2 baseline is 63.67%.
Combined losses; DINOv2 baseline is 58.10%.
Unsupervised object discovery; DINOv2 baseline is 58.6.
Focus-oriented retrieval; DINOv2 baseline is 55.2%.
Combined losses; DINOv2 baseline is 81.7%.
What the paper found
DINOv2 can learn useful visual features without labels, but its attention heads often focus on similar or ambiguous regions. This paper introduces two training losses to encourage region-to-focus correspondence: an inside-focus loss matches attention to a randomly cropped region, while an outside-focus loss matches attention beyond a masked region. Both use prompt-guided attention head selection to steer selected heads, requiring no object annotations. Trained on ImageNet-1k with ViT-L/16-Reg, the combined losses produce spatially specialized heads whose individual maps are more concentrated while their combined attention covers a wider area. On zero-shot segmentation, ImageNet-Segmentation mIoU rises from 63.67% to 67.29%, and PASCAL VOC mIoU rises from 58.10% to 60.68%. Unsupervised object discovery on VOC07 improves from 58.6 to 61.6 CorLoc, while focus-oriented COCO retrieval with a visual prompt reaches 55.9% MP@100, versus 55.2% for DINOv2. The trade-off is weaker global classification: ImageNet-1k k-NN accuracy falls from 81.7% to 80.3%. The study also compares attention maps from OpenAI's CLIP and other pretrained models, but its central contribution is showing that paired crop-based and mask-based losses can make self-supervised ViT attention more spatially diverse and useful for dense visual tasks.
Original abstract
Self-supervised learning in Vision Transformer models, such as DINO, enables robust visual perception without requiring labeled data by leveraging large-scale image datasets. However, these models lack a focusing mechanism during training, resulting in perceptual representations that are often ambiguous or overly similar. In this work, we introduce two visual focusing loss functions designed to establish correspondence between the model's attention and specific regions within images. Specifically, we leverage the multi-head attention mechanism within the model to selectively steer the model's focus, leading to the emergence of spatial attention head specialization, characterized by diverse and spatially focused perceptual capabilities without requiring annotations. Through both qualitative and quantitative evaluations, we demonstrate that our method substantially increases the spatial selectivity and diversity of attention heads, improving both interpretability and performance on dense prediction tasks such as zero-shot image segmentation.
Read the original paperMore in Attention Mechanisms
Browse all 27 papers →Hard Sample-Conditioned Decorrelated Attention for Robust Face Recognition
Tahar Chettaoui, Guray Ozgur, Eduarda Caldeira, Naser Damer, Fadi Boutros
ViT-HarDis encourages attention heads to specialize more on difficult face samples, improving recognition without extra inference cost.
Muir: Linear Attention with Sensitivity-Allocated Gating and Matched-Filter Readout
Tushaar Gangavarapu, Gil Tabak, Clemens JS Schaefer
Muir redesigns linear attention’s gates and readout to improve language-model quality and long-context retrieval while preserving efficient training and inference.
HyScale: Scaling Long-Context Training for Learned Sparse Attention
Yangshen Deng, Xinjing Huang, Yulong Wang, Yunzhuo Liu, Jun Hu, Shuyong Tan, Chun-Chih Tseng, Mingji Han, Ray Ying, Key Zhang, Luo Mai
HyScale makes training long-context sparse attention faster by avoiding large selection tables and reusing selection results during recomputation.