Rethinking Attention Locality in Spiking Transformers

Rethinking Attention Locality in Spiking Transformers argues that making attention computationally cheaper does not guarantee spatially local token interactions. In Softmax-free Spiking Self-Attention, offset-based grouping in LSSA can connect distant pixels, while uniformly applying LRF-SSA produces inconsistent Mean Attention Distance across layers and architectures. The paper introduces Spatially Contiguous Local Attention with Boundary Continuity Pathway, or SCLA-BCP: SCLA partitions feature maps into non-overlapping, spatially contiguous regions and performs attention within each region, while BCP uses a lightweight 3×3 depth-wise convolution, batch normalization, and max-pooling pathway to exchange information across region boundaries. A hierarchical deployment strategy applies SCLA-BCP to the first half of SSA layers in ViT-like models and to Stages 1–2 in hierarchical models. Across 7 static and neuromorphic datasets, including ImageNet-1K, CIFAR10-DVS, N-Caltech101, Microsoft COCO 2017, and ADE20K, the method consistently improves classification, detection, and segmentation. On ImageNet-1K, Spikingformer reaches 78.62% top-1 accuracy versus 77.64% for its baseline; on Microsoft COCO 2017, it improves mAP@50 by up to 9.50%; and on ADE20K, it improves mIoU by up to 3.42%. The gains require limited overhead, including just 0.03M additional parameters in the Spikingformer ImageNet-1K configuration, while MAD measurements and ablations confirm that contiguous regional attention, combined with boundary compensation, produces genuinely more localized interactions.