Kernel Renormalization in Bayesian Deep Neural Networks: the Equivalent Wishart Ansatz in the Proportional Regime

This paper, from INFN and the University of Parma, develops an approximate statistical-mechanics theory for Bayesian deep neural networks in the proportional regime where sample size P and width N grow together. The key novelty is the Equivalent Wishart Ansatz, which treats the hierarchical empirical kernels of multilayer perceptrons as if their dominant finite-width fluctuations were Wishart-distributed, collapsing an intractable matrix-valued problem into at most L scalar order parameters. That reduction yields a large-deviation effective action for the posterior and a renormalized neural-network Gaussian process kernel, so learning curves can be predicted from a low-dimensional theory rather than P×P kernel self-consistency equations. The authors extend the construction to nonzero-mean activations such as ReLU via a noncentral Wishart analysis, and to convolutional networks through a stacked kernel renormalization scheme that captures patch–patch correlations. They validate the theory against Bayesian posterior sampling with Langevin Monte Carlo, MALA, pCN, and NUTS HMC on MNIST, CIFAR-10, Gaussian teacher-student data, and networks with depths up to L≈10 and P≈10^3, finding excellent agreement except for two systematic breakdowns at large depth and load α=P/N, including a newly reported metastable posterior transition around L>5. In the mean-field/μP parametrization, the theory shows that improved generalization is driven mainly by variance suppression rather than bias change, while the effective kernel remains only globally rescaled. Overall, the result reframes finite-width deep learning as a renormalized-kernel problem governed by sample-to-width scaling, not just lazy versus rich training.