Graph Machine: Towards Better Pretraining via Edges

Graph Machine, or GM, proposes a fourth sequence-modeling regime: it keeps an O(n) state like a Transformer but uses dynamically addressed sparse access instead of scanning the full history. Its state stores node features plus differentiable, pointer-like edge indices and weights. A referral mechanism composes multi-hop neighborhoods by approximating sparse adjacency-matrix powers, while Sparse Edge Attention combines edge weights with query–key scores as a product of experts. In a controlled pretraining study, researchers replaced 75% of the dense Transformer layers in Qwen3-0.6B with GM layers and trained from scratch on 15.7B tokens from FineWeb-Edu. At a 4,096-token context, retrieving only 2 or 4 positions per KV head corresponds to 0.098% or 0.195% of dense causal KV access. The strongest configuration, Hyperion-K16-R3-S, reduced final test loss by 0.003 relative to Qwen3 while using 19% less referral-plus-attention compute; the result suggests that most dense attention can be replaced by learned relational traversal at this scale. The main caveat is systems efficiency: the prototype was several times slower than Qwen3 on an H100, so custom kernels and larger-scale evaluations remain necessary.