Speed Limit for Information Acquisition in Stochastic Learning Dynamics

This paper treats stochastic gradient descent as a Markovian stochastic process and introduces a Fisher-information speed limit for how quickly model parameters can encode latent variables from the data-generating process. The central inequality bounds Fisher-information flow by two information budgets: a drift term from the latent-variable sensitivity of the mean gradient, and a noise term from latent-variable dependence in mini-batch gradient covariance. In basis-function linear regression, the dynamics reduce near convergence to independent Ornstein–Uhlenbeck modes, showing that latent information is acquired in an order set by the relaxation eigenvalues, while each latent variable’s acquisition profile depends on how strongly it couples to those modes. A single-parameter Gaussian linear-regression experiment uses m=100, H=1, σ=0.5, and ε=0.01; the measured Fisher-flow peak occurs at τ=4.040, while the drift-budget relaxation time is τdrift=4.033, demonstrating a nearly tight bound. The study also models the nonlinear target y=A sin(ωx+ϕ)+η with Gaussian radial basis functions using d=12 and l=0.25, resolving distinct acquisition times for amplitude, frequency, and phase. The framework therefore diagnoses not just whether learning succeeds, but when specific aspects of the underlying data-generating mechanism become statistically recoverable from parameters, with possible extensions to nonlinear networks and heavy-tailed, non-Gaussian gradient noise.