The Implicit Bias of Depth: From Neural Collapse to Softmax Codes

The paper studies the deep unconstrained feature model under multiclass cross-entropy without explicit regularization, isolating the implicit bias of gradient flow in deep linear classifiers. Its main result is that depth itself induces a low-rank bias that competes with neural collapse: for L=2 and K≥6, or L≥3 and K≥4, the asymptotic max-margin landscape is no longer benign and deep neural collapse is not globally optimal, even though it remains a local optimum. The global optimum in the large-depth limit is a rank-2 softmax code, where class means lie uniformly on a circle, matching the d=2 bottleneck solution of Jiang et al. (2023). The mechanism is spectral: low-rank matrices propagate norm more efficiently through repeated products, so they achieve larger logits and smaller loss than the simplex ETF geometry associated with neural collapse. Dynamically, under Hadamard initialization the singular-value flow reduces to explicit ODEs showing a rich-get-richer effect, where larger singular modes grow faster near the origin, making neural collapse unstable for L>1 in the linearized regime. This instability grows with depth, while random Gaussian initialization at large width counteracts it by biasing early motion toward neural collapse through concentration of measure; the paper proves dZ/dt at time 0 converges in probability to (L+1)ϵ^{2L}(S⊗1_n^T). Experiments on deep UFMs and ResNet-20 heads trained on MNIST and CIFAR-10 confirm that increasing depth lowers effective rank, increasing width raises it, and lower-rank solutions tend to generalize worse.