NTH

Training Vision Transformers to Focus: Emergence of Spatial Attention Head Specialization

AuthorsYuji Nozawa, Yu-Chieh Lin, Youyang Ng

AffiliationsKioxia Corporation

November 15, 2026 2 min read
Watch on YouTube
The one-line take

The method encourages Vision Transformers to develop distinct, spatially focused attention heads without image annotations.

Key results

67.29%
ImageNet-Segmentation mIoU

Combined losses; DINOv2 baseline is 63.67%.

60.68%
PASCAL VOC mIoU

Combined losses; DINOv2 baseline is 58.10%.

61.6
VOC07 CorLoc

Unsupervised object discovery; DINOv2 baseline is 58.6.

55.9%
COCO visual-prompt retrieval MP@100

Focus-oriented retrieval; DINOv2 baseline is 55.2%.

80.3%
ImageNet-1k k-NN accuracy

Combined losses; DINOv2 baseline is 81.7%.

What the paper found

DINOv2 can learn useful visual features without labels, but its attention heads often focus on similar or ambiguous regions. This paper introduces two training losses to encourage region-to-focus correspondence: an inside-focus loss matches attention to a randomly cropped region, while an outside-focus loss matches attention beyond a masked region. Both use prompt-guided attention head selection to steer selected heads, requiring no object annotations. Trained on ImageNet-1k with ViT-L/16-Reg, the combined losses produce spatially specialized heads whose individual maps are more concentrated while their combined attention covers a wider area. On zero-shot segmentation, ImageNet-Segmentation mIoU rises from 63.67% to 67.29%, and PASCAL VOC mIoU rises from 58.10% to 60.68%. Unsupervised object discovery on VOC07 improves from 58.6 to 61.6 CorLoc, while focus-oriented COCO retrieval with a visual prompt reaches 55.9% MP@100, versus 55.2% for DINOv2. The trade-off is weaker global classification: ImageNet-1k k-NN accuracy falls from 81.7% to 80.3%. The study also compares attention maps from OpenAI's CLIP and other pretrained models, but its central contribution is showing that paired crop-based and mask-based losses can make self-supervised ViT attention more spatially diverse and useful for dense visual tasks.

Original abstract

Self-supervised learning in Vision Transformer models, such as DINO, enables robust visual perception without requiring labeled data by leveraging large-scale image datasets. However, these models lack a focusing mechanism during training, resulting in perceptual representations that are often ambiguous or overly similar. In this work, we introduce two visual focusing loss functions designed to establish correspondence between the model's attention and specific regions within images. Specifically, we leverage the multi-head attention mechanism within the model to selectively steer the model's focus, leading to the emergence of spatial attention head specialization, characterized by diverse and spatially focused perceptual capabilities without requiring annotations. Through both qualitative and quantitative evaluations, we demonstrate that our method substantially increases the spatial selectivity and diversity of attention heads, improving both interpretability and performance on dense prediction tasks such as zero-shot image segmentation.

Read the original paper

More in Attention Mechanisms

Browse all 27 papers →
03Attention

HyScale: Scaling Long-Context Training for Learned Sparse Attention

Yangshen Deng, Xinjing Huang, Yulong Wang, Yunzhuo Liu, Jun Hu, Shuyong Tan, Chun-Chih Tseng, Mingji Han, Ray Ying, Key Zhang, Luo Mai

HyScale makes training long-context sparse attention faster by avoiding large selection tables and reusing selection results during recomputation.

Read analysis