Convergence of the Momentum Method for Semialgebraic Functions with Locally Lipschitz Gradients
From MaRDI portal
(Redirected from Publication:6071885)
Abstract: We propose a new length formula that governs the iterates of the momentum method when minimizing differentiable semi-algebraic functions with locally Lipschitz gradients. It enables us to establish local convergence, global convergence, and convergence to local minimizers without assuming global Lipschitz continuity of the gradient, coercivity, and a global growth condition, as is done in the literature. As a result, we provide the first convergence guarantee of the momentum method starting from arbitrary initial points when applied to principal component analysis, matrix sensing, and linear neural networks.
Recommendations
- Convergence of gradient algorithms for nonconvex \(C^{1+ \alpha}\) cost functions
- Semistability of steepest descent with momentum for quadratic functions
- Generalized momentum-based methods: a Hamiltonian perspective
- On the influence of momentum acceleration on online learning
- Continuous time analysis of momentum methods
- A study on the optimal double parameters for steepest descent with momentum
- A Diffusion Approximation Theory of Momentum Stochastic Gradient Descent in Nonconvex Optimization
- Stochastic momentum methods for non-convex learning without bounded assumptions
- scientific article; zbMATH DE number 165427
- Backtracking gradient descent method and some applications in large scale optimisation. II: Algorithms and experiments
Cites work
- Accelerated gradient methods for nonconvex nonlinear and stochastic programming
- An inexact proximal gradient algorithm with extrapolation for a class of nonconvex nonsmooth optimization problems
- Behavior of accelerated gradient methods near critical points of nonconvex functions
- Calculus of the exponent of Kurdyka-Łojasiewicz inequality and its applications to linear convergence of first-order methods
- Certifying the Absence of Spurious Local Minima at Infinity
- Clarke Subgradients of Stratifiable Functions
- Continuous time analysis of momentum methods
- Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward-backward splitting, and regularized Gauss-Seidel methods
- Convergence of the Iterates of Descent Methods for Analytic Cost Functions
- Correction to: ``Global convergence of the gradient method for functions definable in o-minimal structures
- Decoding by Linear Programming
- First-order methods almost always avoid strict saddle points
- Genericity in polynomial optimization
- Heavy-ball method in nonconvex optimization problems
- scientific article; zbMATH DE number 4029737 (Why is no real title available?)
- scientific article; zbMATH DE number 46303 (Why is no real title available?)
- scientific article; zbMATH DE number 1160037 (Why is no real title available?)
- scientific article; zbMATH DE number 1461253 (Why is no real title available?)
- scientific article; zbMATH DE number 3109695 (Why is no real title available?)
- iPiano: inertial proximal algorithm for nonconvex optimization
- Lectures on convex optimization
- Linear convergence of proximal gradient algorithm with extrapolation for a class of nonconvex nonsmooth minimization problems
- Local convergence of the heavy-ball method and iPiano for non-convex optimization
- Mathematical analysis II. Transl. from the 4th Russian edition by Roger Cooke
- Nonconvergence to unstable points in urn models and stochastic approximations
- On gradients of functions definable in o-minimal structures
- Proximal alternating linearized minimization for nonconvex and nonsmooth problems
- Some methods of speeding up the convergence of iteration methods
- Submersions and preimages of sets of measure zero
- Variational Analysis in Sobolev and BV Spaces
Cited in
(4)- Global convergence of the gradient method for functions definable in o-minimal structures
- Local convergence analysis for nonisolated solutions to derivative-free methods of optimization
- A normal map-based proximal stochastic gradient method: convergence and identification properties
- Globally convergent derivative-free methods in nonconvex optimization with and without noise
This page was built for publication: Convergence of the Momentum Method for Semialgebraic Functions with Locally Lipschitz Gradients
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6071885)
