The Devil is in the Condition Numbers: Why is GLU Better than non-GLU Structure?

This paper explains why GLU and its variants, especially ReGLU and SwiGLU, outperform non-gated feedforward blocks by tracing the effect to NTK conditioning rather than to a smaller generalization gap. In a two-layer network analyzed in the neural tangent kernel regime, the authors derive an approximate factorization K̃ ≈ K ⊙ (XX⊤/d), showing that the GLU gate suppresses gradient correlation and makes the kernel spectrum more compact. Under Gaussian inputs and LeCun initialization, they prove that ReGLU changes the scaling of the largest NTK eigenvalue from Θ(mn) for ReLU to Θ(mn/d), while the smallest eigenvalue remains Θ(m) but becomes larger than the non-GLU case; this yields a smaller condition number and therefore faster gradient descent convergence. Their experiments confirm the spectral prediction on synthetic data and on real models: replacing FFN blocks in ViT and GPT-2 with GLU variants reduces NTK condition numbers across ReLU, GELU/GEGLU, and SiLU/SwiGLU settings. The paper also derives a loss-crossing phenomenon: non-GLU models can descend faster early in training, but GLU overtakes later because its larger λmin controls the slowest mode. In contrast, across MLP Mixer on CIFAR-10, ViT on Tiny ImageNet, and GPT-2 on FineWeb-Edu, the generalization gap Ltest−Ltrain remains statistically similar with and without GLU, with permutation-test p-values mostly above 0.05, indicating that GLU’s primary benefit is optimization speed, not gap reduction.