Distance generalization in transformers: why bother with positional encoding?

This paper isolates distance generalization: whether a transformer can copy tokens when the source-to-recall delay changes, while the overall context window stays fixed, separating unseen token dependencies from unseen positions in length generalization. Experiments use full and selective delay-copy tasks with a causal decoder-only transformer containing 8 layers, 8 attention heads, and hidden size 512, trained on 10M synthetic tokens with a fixed context length of 256. Models trained on delays from 15 to 25 show a counterintuitive ranking: NoPE, with no explicit positional encoding, usually generalizes better to unseen distances than ALiBi and RoPE, even though RoPE is widely used in models such as Llama 3 and DeepSeek-V3. The result suggests causal attention can infer relative position dynamically, although NoPE becomes unreliable in smaller models. Increasing training diversity across delay distances improves absolute out-of-distribution accuracy, but with diminishing relative returns. Transfer learning between full and selective copying is conditional: NoPE and ALiBi often benefit at moderate extrapolation distances but can suffer at larger ones, while RoPE generally exhibits weaker transfer and, in some settings, negative interference. The study argues that distance generalization should be evaluated separately from context-length extrapolation and that positional encoding choices require reconsideration.