Neural Quadratic Forms: A Unified Minimal Model for Sudden Learning and Scaling Laws

“Neural Quadratic Forms” proposes a single local model for sudden learning and smooth training-time scaling laws. The key result is that permutation symmetry among interchangeable units, smoothness, and zero gradient at zero force a near-initialization expansion of the form f_x(W)−f_x(0)=μᵀg(x)+Tr[WWᵀA(x)]+O(||W||³), where architecture-specific details are compressed into the structure matrix A(x). Two-layer MLPs, CNNs, mixture-of-experts layers, attention heads, matrix-sensing models, and 17 previously separate solvable proxies become instances of this neural quadratic form. For multi-head attention, the leading term depends on value and readout weights, while query and key weights first appear at quartic order, a result relevant to transformer systems such as ChatGPT but not presented as a ChatGPT benchmark. Training closes on the low-dimensional order parameters M=WWᵀ and μ; under a shared eigenbasis for the data matrices, the feature eigenvalues follow a generalized Lotka–Volterra equation. Each mode activates at a characteristic time proportional to (1/ζ)ln(1/ε), so smaller initialization separates learning events into plateaus, while power-law spectra aggregate them into predictable power-law loss curves. Experiments show close agreement across gradient descent, Polyak momentum, and Adam at initialization scales σ=0.01 and σ=0.2, use a synthetic feature-wise setup with m=1000, and reproduce matching dynamics on a 4000-sample MNIST odd-versus-even task. The theory is limited by small initialization, smooth activations, and the assumption of power-law spectra.