Dataset Complexity Shapes Finite-Distance Loss Geometry in Neural Networks

This paper shows that dataset structure, not merely dataset size or input statistics, reshapes the finite-distance geometry of neural-network loss landscapes. The researchers measure multiscale local label mixing with CMS and estimate Franz–Parisi-inspired local entropy around zero-training-error solutions using adaptive sequential Monte Carlo. In a controlled synthetic experiment with 512 points and neighborhoods extending to kmax = 22, increasing label interweaving causes a sharper, earlier contraction of the effective low-loss solution volume near the reference; farther away, the radial loss-entropy derivative becomes weak and nearly common across conditions. The networks use two tanh hidden layers and contain 2545 trainable parameters for synthetic data and 2461 for downsampled MNIST. On MNIST, label randomization amplifies the bottleneck: at radius r = 1, shell-weighted training accuracy falls to 0.598 for the most complex synthetic condition compared with about 0.93 for low complexity, while natural digit-pair tasks show a milder but consistent trend, spanning CMS values from approximately 0.013 to 0.277 and accuracies of 0.979 for 0/1 versus 0.893 for 4/9. A symmetric finite-distance check also finds hardening of 0.417 for clean labels versus 3.508 for randomized labels, ruling out residual first-order tilt as the sole explanation. The central result is that complexity changes where solution neighborhoods contract and can alter their radial shape, something a single Hessian-based flatness measure cannot capture.