When Attention Goes Blind: Numerical Failure in ALiBi Positional Encodings

The paper identifies a numerical failure in ALiBi, the parameter-free positional encoding used by models such as BLOOM, Falcon-RW, and MPT: its linearly increasing attention bias eventually pushes softmax exponentials below floating-point range, turning distant attention weights into exact zeros and making heads partially blind. In bfloat16, 36.6% of attention entries have underflowed by token distance 2048. The authors confirm this behavior in pretrained BLOOM 560M, Falcon-RW 7B, and MPT 7B, then isolate its effects with 148M-parameter Llama decoder models trained on 20B FineWeb-Edu tokens. Standard commonsense, question-answering, and language benchmarks change only modestly, but associative retrieval is much more sensitive: default ALiBi remains strong on needle-in-a-haystack retrieval while suffering on out-of-context passkey retrieval. Four training-time mitigations are tested—bias clamping, redesigned slopes, logarithmic distances, and soft capping. The strongest passkey result comes from combining clamping with log-scaled distances, raising out-of-context passkey AUC to 0.79 versus 0.08 for the ALiBi baseline. However, no mitigation dominates every retrieval task, and several reduce needle-in-a-haystack performance. The authors recommend clamping or explicit sliding-window attention when a bounded receptive field is intended, and view log-scaled distances as the most consistent option for passkey extrapolation.