Inertial Newton algorithms avoiding strict saddle points

Abstract: We study the asymptotic behavior of second-order algorithms mixing Newton's method and inertial gradient descent in non-convex landscapes. We show that, despite the Newtonian behavior of these methods, they almost always escape strict saddle points. We also evidence the role played by the hyper-parameters of these methods in their qualitative behavior near critical points. The theoretical results are supported by numerical illustrations.

Recommendations

Cites work

scientific article; zbMATH DE number 3163943 (Why is no real title available?)
scientific article; zbMATH DE number 49110 (Why is no real title available?)
scientific article; zbMATH DE number 3479763 (Why is no real title available?)
scientific article; zbMATH DE number 3213086 (Why is no real title available?)
scientific article; zbMATH DE number 5060482 (Why is no real title available?)
A Lemma in the Theory of Structural Stability of Differential Equations
A New Value Iteration method for the Average Cost Dynamic Programming Problem
A dynamical approach to an inertial forward-backward algorithm for convex minimization
A generalization of Hartman's linearization theorem
A gradient-type algorithm with backward inertial steps associated to a nonconvex minimization problem
A second-order gradient-like dissipative dynamical system with Hessian-driven damping. Application to optimization and mechanics.
An extension of the second order dynamical system that models Nesterov's convex gradient method
An inertial Newton algorithm for deep learning
An inertial forward-backward algorithm for the minimization of the sum of two nonconvex functions
Backtracking gradient descent method and some applications in large scale optimisation. II: Algorithms and experiments
Behavior of accelerated gradient methods near critical points of nonconvex functions
Continuous Newton-like inertial dynamics for monotone inclusions
Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward-backward splitting, and regularized Gauss-Seidel methods
Convergence rates for an inertial algorithm of gradient type associated to a smooth non-convex minimization
Fast convergence of inertial dynamics and algorithms with asymptotic vanishing viscosity
Fast convex optimization via inertial dynamics with Hessian driven damping
Fast optimization via inertial dynamics with closed-loop damping
First-order optimization algorithms via inertial systems with Hessian driven damping
Gradient descent only converges to minimizers: non-isolated critical points and invariant regions
Limit Points of Sequences in Metric Spaces
Local convergence of the heavy-ball method and iPiano for non-convex optimization
Newton-like inertial dynamics and proximal algorithms governed by maximally monotone operators
Optimal convergence rates for Nesterov acceleration
Rate of convergence of the Nesterov accelerated gradient method in the subcritical case α ≤ 3
Some methods of speeding up the convergence of iteration methods
The Differential Inclusion Modeling FISTA Algorithm and Optimality of Convergence Rate in the Case b $\leq3$
The gradient and heavy ball with friction dynamical systems: The quasiconvex case
The second-order in time continuous Newton method
The stable, center-stable, center, center-unstable, unstable manifolds
Tikhonov regularization of a second order dynamical system with Hessian driven damping
Understanding the acceleration phenomenon via high-resolution differential equations

Cited in

(1)

A fast and simple modification of Newton's method avoiding saddle points