Ask Self, Ask Others: Relation Is All You Need

This paper proposes Relation, a token-mixing primitive that changes the Transformer order from scores directly determining attention flow to relation being constructed first and flow derived afterward. Its Self–Exchange Relation separates each token’s self-evidence from exchange evidence with prior tokens: Self uses a bounded sigmoid mapping, Exchange uses SiLU, and a causal Relation matrix is normalized only after a logarithmic count correction. Multi-Head Relation adds alternating adjacent-head Givens rotations to mix information states. In matched decoder-only models trained on TinyStories and the SmolLM corpus at 10M, 30M, and 100M parameters, Full Relation reduces final validation NLL versus MHA by 0.0412, 0.0151, and 0.0310, respectively. FlashRelation is an exact tiled implementation that avoids materializing the full relation and flow matrices; at context length 1024 it is 3.60–4.41× faster than the materialized reference and reaches 76.4–84.9% of PyTorch FlashAttention throughput on scale-matched workloads using an NVIDIA RTX 5090. Linear Relation replaces token-wise history with a recurrent state, reducing token-mixing complexity to O(Td²/H) and enabling fixed-size decode history, while Hybrid Relation combines nine Linear Relation layers with three Full Relation layers—75% Linear Relation—and achieves an NLL of 1.2780 in a 31.97M-parameter model. The result is a relation-first alternative to standard attention, complementary to systems such as FlashAttention and architectures such as Kimi Linear, though experiments remain limited to models up to approximately 100M parameters.