Rigorous dynamical mean field theory for stochastic gradient descent methods
From MaRDI portal
(Redirected from Publication:6413773)
Abstract: We prove closed-form equations for the exact high-dimensional asymptotics of a family of first order gradient-based methods, learning an estimator (e.g. M-estimator, shallow neural network, ...) from observations on Gaussian data with empirical risk minimization. This includes widely used algorithms such as stochastic gradient descent (SGD) or Nesterov acceleration. The obtained equations match those resulting from the discretization of dynamical mean-field theory (DMFT) equations from statistical physics when applied to gradient flow. Our proof method allows us to give an explicit description of how memory kernels build up in the effective dynamics, and to include non-separable update functions, allowing datasets with non-identity covariance matrices. Finally, we provide numerical implementations of the equations for SGD with generic extensive batch-size and with constant learning rates.
Cited in
(11)- Linear operator approximate message passing (OpAMP)
- High-dimensional scaling limits and fluctuations of online least-squares SGD with smooth covariance
- Entrywise dynamics and universality of general first order methods
- High-dimensional learning of narrow neural networks
- Hitting the high-dimensional notes: an ODE for SGD learning dynamics on GLMs and multi-index models
- Differentially private learning beyond the classical dimensionality regime
- Asymptotic dynamics of alternating minimization for bilinear regression
- Gradient descent inference in empirical risk minimization
- Uncertainty quantification for iterative algorithms in linear models with application to early stopping
- Stochastic gradient descent in high dimensions for multi-spiked tensor PCA
- Langevin dynamics for high-dimensional optimization: the case of multi-spiked tensor PCA
This page was built for publication: Rigorous dynamical mean field theory for stochastic gradient descent methods
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6413773)
