On the diffusion approximation of nonconvex stochastic gradient descent
From MaRDI portal
Abstract: We study the Stochastic Gradient Descent (SGD) method in nonconvex optimization problems from the point of view of approximating diffusion processes. We prove rigorously that the diffusion process can approximate the SGD algorithm weakly using the weak form of master equation for probability evolution. In the small step size regime and the presence of omnidirectional noise, our weak approximating diffusion process suggests the following dynamics for the SGD iteration starting from a local minimizer (resp.~saddle point): it escapes in a number of iterations exponentially (resp.~almost linearly) dependent on the inverse stepsize. The results are obtained using the theory for random perturbations of dynamical systems (theory of large deviations for local minimizers and theory of exiting for unstable stationary points). In addition, we discuss the effects of batch size for the deep neural networks, and we find that small batch size is helpful for SGD algorithms to escape unstable stationary points and sharp minimizers. Our theory indicates that one should increase the batch size at later stage for the SGD to be trapped in flat minimizers for better generalization.
Recommendations
- A Diffusion Approximation Theory of Momentum Stochastic Gradient Descent in Nonconvex Optimization
- Uniform-in-time weak error analysis for stochastic gradient descent algorithms via diffusion approximation
- On large batch training and sharp minima: a Fokker-Planck perspective
- Convergence of constant step stochastic gradient descent for non-smooth non-convex functions
- Analysis of stochastic gradient descent in continuous time
Cited in
(27)- A Diffusion Approximation Theory of Momentum Stochastic Gradient Descent in Nonconvex Optimization
- Momentum-based accelerated mirror descent stochastic approximation for robust topology optimization under stochastic loads
- A probability approximation framework: Markov process approach
- On large batch training and sharp minima: a Fokker-Planck perspective
- Random batch methods (RBM) for interacting particle systems
- Multistart algorithm for identifying all optima of nonconvex stochastic functions
- Uniform-in-time weak error analysis for stochastic gradient descent algorithms via diffusion approximation
- A random batch method for efficient ensemble forecasts of multiscale turbulent systems
- Analysis of stochastic gradient descent in continuous time
- Approximation to stochastic variance reduced gradient Langevin dynamics by stochastic delay differential equations
- A Continuous-Time Analysis of Distributed Stochastic Gradient
- Numerical analysis for inchworm Monte Carlo method: Sign problem and error growth
- On a stochastic differential equation with correction term governed by a monotone and Lipschitz continuous operator
- Applications of Fokker Planck equations in machine learning algorithms
- One-dimensional system arising in stochastic gradient descent
- A convergence analysis of the perturbed compositional gradient flow: averaging principle and normal deviations
- Large deviation upper bounds and improved MSE rates of nonlinear SGD: heavy-tailed noise and power of symmetry
- On uniform-in-time diffusion approximation for stochastic gradient descent
- The effective noise of stochastic gradient descent
- Stochastic dual dynamical systems for linear equality constrained convex optimization problems
- Stochastic modified equations and dynamics of stochastic gradient algorithms. I: Mathematical foundations
- Stochastic modified flows for Riemannian stochastic gradient descent
- Stochastic differential equation approximations of generative adversarial network training and its long-run behavior
- On the fast convergence of random perturbations of the gradient flow
- A Random-Batch Monte Carlo Method for Many-Body Systems with Singular Kernels
- Second-Order Guarantees of Stochastic Gradient Descent in Nonconvex Optimization
- Convergence of Markov chains for constant step-size stochastic gradient descent with separable functions
This page was built for publication: On the diffusion approximation of nonconvex stochastic gradient descent
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q1734292)
