Gradient Descent Only Converges to Minimizers: Non-Isolated Critical Points and Invariant Regions
From MaRDI portal
Publication:4638050
DOI10.4230/LIPIcs.ITCS.2017.2zbMath1402.90210arXiv1605.00405MaRDI QIDQ4638050
Georgios Piliouras, Ioannis Panageas
Publication date: 3 May 2018
Full work available at URL: https://arxiv.org/abs/1605.00405
Related Items
Analysis of Asymptotic Escape of Strict Saddle Sets in Manifold Optimization ⋮ Model-free Nonconvex Matrix Completion: Local Minima Analysis and Applications in Memory-efficient Kernel PCA ⋮ Convergence guarantees for a class of non-convex and non-smooth optimization problems ⋮ On initial point selection of the steepest descent algorithm for general quadratic functions ⋮ Proximal methods avoid active strict saddles of weakly convex functions ⋮ Landscape analysis for shallow neural networks: complete classification of critical points for affine target functions ⋮ Unnamed Item ⋮ A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functions ⋮ Global convergence of the gradient method for functions definable in o-minimal structures ⋮ Polynomial‐time universality and limitations of deep learning ⋮ Statistical Inference with Local Optima ⋮ A geometric approach of gradient descent algorithms in linear neural networks ⋮ Run-and-inspect method for nonconvex optimization and global optimality bounds for R-local minimizers ⋮ First-order methods almost always avoid strict saddle points ⋮ Sufficient Conditions for Instability of the Subgradient Method with Constant Step Size ⋮ Inertial Newton algorithms avoiding strict saddle points ⋮ A Newton-Based Method for Nonconvex Optimization with Fast Evasion of Saddle Points ⋮ Stochastic optimization with momentum: convergence, fluctuations, and traps avoidance ⋮ Backtracking gradient descent method and some applications in large scale optimisation. II: Algorithms and experiments ⋮ Unnamed Item ⋮ Unnamed Item ⋮ A geometric analysis of phase retrieval ⋮ Multiscale sparse microcanonical models ⋮ Mutation, Sexual Reproduction and Survival in Dynamic Environments ⋮ Gradient Descent Only Converges to Minimizers: Non-Isolated Critical Points and Invariant Regions ⋮ Null space gradient flows for constrained optimization with applications to shape optimization ⋮ Extending the Step-Size Restriction for Gradient Descent to Avoid Strict Saddle Points ⋮ On Gradient-Based Learning in Continuous Games
Cites Work
- Unnamed Item
- Unnamed Item
- Unnamed Item
- Unnamed Item
- Unnamed Item
- Nonconvergence to unstable points in urn models and stochastic approximations
- Introductory lectures on convex optimization. A basic course.
- Cubic regularization of Newton method and its global performance
- Algorithms, games, and evolution
- Phase Retrieval via Wirtinger Flow: Theory and Algorithms
- Complete Dictionary Recovery Over the Sphere II: Recovery by Riemannian Trust-Region Method
- On the use of directions of negative curvature in a modified newton method
- Trust Region Methods
- Gradient Descent Only Converges to Minimizers: Non-Isolated Critical Points and Invariant Regions
- Mutation, Sexual Reproduction and Survival in Dynamic Environments
- Multiplicative updates outperform generic no-regret learning in congestion games
