Second-Order Guarantees of Stochastic Gradient Descent in Nonconvex Optimization
From MaRDI portal
Abstract: Recent years have seen increased interest in performance guarantees of gradient descent algorithms for non-convex optimization. A number of works have uncovered that gradient noise plays a critical role in the ability of gradient descent recursions to efficiently escape saddle-points and reach second-order stationary points. Most available works limit the gradient noise component to be bounded with probability one or sub-Gaussian and leverage concentration inequalities to arrive at high-probability results. We present an alternate approach, relying primarily on mean-square arguments and show that a more relaxed relative bound on the gradient noise variance is sufficient to ensure efficient escape from saddle-points without the need to inject additional noise, employ alternating step-sizes or rely on a global dispersive noise assumption, as long as a gradient noise component is present in a descent direction for every saddle-point.
Recommendations
- Second-order guarantees in centralized, federated and decentralized nonconvex optimization
- Inexact proximal stochastic second-order methods for nonconvex composite optimization
- A second-order gradient method for convex minimization
- Convergence rates for the stochastic gradient descent method for non-convex objective functions
- Second-Order Online Nonconvex Optimization
- On the diffusion approximation of nonconvex stochastic gradient descent
- Stochastic generalized gradient method for nonconvex nonsmooth stochastic optimization
- Approaching nonsmooth nonconvex minimization through second-order proximal-gradient dynamical systems
- Second-order guarantees of distributed gradient algorithms
Cited in
(3)
This page was built for publication: Second-Order Guarantees of Stochastic Gradient Descent in Nonconvex Optimization
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6053157)
