Your Transformer Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMs

This paper argues that ordinary decoder-only Transformers exhibit a measurable Superposition Linearity Hypothesis: when token embeddings from two unrelated text streams are averaged position by position, a single forward pass retains information from both, rather than producing semantic collapse. Across Pythia, Llama, Qwen, OLMo, and Gemma models evaluated on TinyStories and FineWeb, each stream’s preferred next token appears in the mixed distribution’s top-10 in roughly 30–40% of cases and reaches 65% recovery by top-100, far above a frequency-only baseline. Training analysis with Pythia checkpoints shows that this behavior is strongest at initialization and degrades during pretraining, suggesting an architectural bias rather than a learned capability; the residual stream’s final third remains especially linear. A lightweight self-distillation objective, using FineWeb pairs and less than 0.025% of the original pretraining-data scale, restores the property: for Pythia-2.8B, mean KL divergence from the ideal average distribution falls from 1.86 to 0.27. The paper also identifies a decoding obstacle: averaging logits behaves like a geometric mean, which can suppress tokens specific to either stream. Its Joint Contrastive decoder uses a smaller guide model to separate the streams, raising LAMBADA accuracy on superposed inputs to 0.430 for Llama-3.2-3B, versus 0.182 for pretrained mixing, though still below the 0.540 single-stream guide baseline. The experiments include OpenAI’s GPT-5.2 as a TinyStories judge, while the proposed approach points toward theoretically 2× inference throughput and reduced KV-cache use, subject to quality and context-length limitations.