Full-bandwidth transformer

The Full-bandwidth transformer widens the narrow vertical feedback channel in autoregressive models: instead of returning only the sampled token, it fuses that token’s embedding with the previous top-layer hidden state using a gated linear unit, then feeds the result into the next decoding step. This lets non-verbalized plans, uncertainty, and intermediate computations receive a fresh traversal through the full stack while preserving the standard architecture, KV cache, language-modeling objective, and Microsoft-compatible serving workflows. The added decoding cost is under 1% per token, requiring only two projection multiplications. To train the recurrence without sacrificing parallel teacher forcing, the method uses scheduled multi-pass training, prefix mixin, RMS normalization, tied input-output embeddings, and hidden-state jitter; notably, adding 3% three-pass batches stabilizes feedback beyond the training horizon. Experiments use 1B-parameter models trained on up to 400B tokens with the Phi-4 data mixture, and optional fused prefilling roughly doubles data efficiency. On the 200B-token model, Soft decoding raises MATH-500 Pass@1 from 0.27 to 0.37, while instruction-tuned GSM8K improves from 64.52 to 67.93 and HumanEval reaches 45.92 Pass@3. Synthetic state-tracking probes show that one recurrent step makes information nearly immediately accessible, reaching 99.6% completion accuracy and 100% delayed-memory accuracy at the input layer. The approach outperforms matched standard baselines and compares favorably with models such as Llama3.2 1B and Qwen3 1.7B, while often producing shorter reasoning traces than conventional chain-of-thought.