Why SGD is not Brownian Motion: A New Perspective on Stochastic Dynamics

This paper argues that stochastic gradient descent should not be modeled as Brownian motion with Gaussian forcing at finite learning rates. Starting from the exact discrete SGD update with sampling with replacement, the authors derive a master equation and a second-order discrete Fokker–Planck expansion, showing that standard Langevin approximations miss terms of order eta squared, including a correction proportional to the squared mean gradient. In a one-dimensional quadratic toy model, this mismatch changes the long-time behavior: the exact discrete dynamics can predict variance growth where the Langevin model predicts a stationary distribution, so the error is qualitative, not just numerical. Near critical points, the theory diagonalizes the covariance dynamics in the mean-Hessian eigenbasis and separates SGD into two regimes: sharp positive-curvature directions confine parameter fluctuations to finite variance, while nearly flat directions remain non-stationary and diffuse, with variance growing roughly linearly in time. Empirically, the paper validates these claims on an MLP with 386 parameters on MNIST, a simplified GPT-style model on Shakespeare, and a 6.6M-parameter NanoGPT trained on WikiText-2. In the NanoGPT experiment, the discrete theory matches the observed variance plateaus across the top 20 Hessian eigendirections, while the standard Langevin prediction underestimates the plateau by about 23 percent at learning rate 0.01. The central novelty is the reinterpretation of SGD as deterministic motion in a fluctuating loss landscape, not a particle driven by external Brownian noise.