Hard Sample-Conditioned Decorrelated Attention for Robust Face Recognition
AuthorsTahar Chettaoui, Guray Ozgur, Eduarda Caldeira, Naser Damer, Fadi Boutros
AffiliationsFraunhofer Institute for Computer Graphics Research IGD, Germany · TU Darmstadt, Germany
ViT-HarDis encourages attention heads to specialize more on difficult face samples, improving recognition without extra inference cost.
Key results
WebFace4M-trained ViT-HarDis result.
WebFace4M-trained ViT-HarDis result.
WebFace4M-trained ViT-HarDis result.
ViT-HarDis result, down from the baseline's 0.40.
Approximate increase; inference cost is unchanged.
What the paper found
ViT-HarDis targets redundant attention heads in Vision Transformers for face recognition by making head decorrelation stronger for identity-ambiguous examples. It measures difficulty by the highest cosine similarity to a competing identity center, excludes misclassified samples from the weighting, and adds the resulting weighted attention-correlation penalty to AdaFace training. Rather than regularizing the whole network, the best configuration applies disentanglement to the final 25% of transformer layers. On WebFace4M, ViT-HarDis achieves 95.84% TAR on IJB-B and 97.24% TAR on IJB-C at FAR 10⁻⁴, and 75.97% Rank-1 on TinyFace, leading the compared Vision Transformer methods on those benchmarks. For the hardest 10% of samples, similarity to the nearest competing identity center falls from 0.40 for the baseline to 0.31, while average inter-head attention similarity drops from 0.21 to 0.01. The approach adds no inference-time cost or architectural changes, but training takes about 30% longer; the reported training comparison used NVIDIA A100 GPUs.
Original abstract
Multi-head attention in Vision Transformers (ViTs) allows independent attention patterns across heads, but without explicit regularization, heads may learn redundant patterns rather than complementary roles. Correlation-based penalties encourage head disentanglement, yet apply uniform regularization across all training samples. In face recognition (FR), hard samples whose embeddings lie close to competing identities are critical for optimization and generalization. Motivated by this, we propose ViT-HarDis, which conditions attention head disentanglement on sample difficulty: samples with high cosine similarity to negative class centers receive stronger head decorrelation, enforcing specialization where identity ambiguity is highest. Experiments show improvements over both the baseline and uniform disentanglement on most benchmarks. Our analysis shows that ViT-HarDis reduces the similarity of hard samples to competing class centers while decreasing inter-head attention similarity, and that applying disentanglement only to the final transformer layers outperforms regularizing the entire network. Trained on WebFace4M, ViT-HarDis achieves SOTA results among ViT-based methods on IJB-B, IJB-C, and TinyFace Rank-1, while achieving the best IJB-B and TinyFace results when trained on MS1MV3, improving FR without architectural modifications or additional inference cost. Code: https://github.com/TaharChettaoui/Hard-Sample-Conditioned-Attention-for-FR.
Read the original paperMore in Attention Mechanisms
Browse all 27 papers →Muir: Linear Attention with Sensitivity-Allocated Gating and Matched-Filter Readout
Tushaar Gangavarapu, Gil Tabak, Clemens JS Schaefer
Muir redesigns linear attention’s gates and readout to improve language-model quality and long-context retrieval while preserving efficient training and inference.
Training Vision Transformers to Focus: Emergence of Spatial Attention Head Specialization
Yuji Nozawa, Yu-Chieh Lin, Youyang Ng
The method encourages Vision Transformers to develop distinct, spatially focused attention heads without image annotations.
HyScale: Scaling Long-Context Training for Learned Sparse Attention
Yangshen Deng, Xinjing Huang, Yulong Wang, Yunzhuo Liu, Jun Hu, Shuyong Tan, Chun-Chih Tseng, Mingji Han, Ray Ying, Key Zhang, Luo Mai
HyScale makes training long-context sparse attention faster by avoiding large selection tables and reusing selection results during recomputation.