Generalized momentum-based methods: a Hamiltonian perspective
From MaRDI portal
Publication:5857293
Abstract: We take a Hamiltonian-based perspective to generalize Nesterov's accelerated gradient descent and Polyak's heavy ball method to a broad class of momentum methods in the setting of (possibly) constrained minimization in Euclidean and non-Euclidean normed vector spaces. Our perspective leads to a generic and unifying nonasymptotic analysis of convergence of these methods in both the function value (in the setting of convex optimization) and in norm of the gradient (in the setting of unconstrained, possibly nonconvex, optimization). Our approach relies upon a time-varying Hamiltonian that produces generalized momentum methods as its equations of motion. The convergence analysis for these methods is intuitive and is based on the conserved quantities of the time-dependent Hamiltonian.
Recommendations
- A variational perspective on accelerated methods in optimization
- Steepest descent with momentum for quadratic functions is a version of the conjugate gradient method
- Momentum and stochastic momentum for stochastic gradient, Newton, proximal point and subspace descent methods
- Nonlinear acceleration of momentum and primal-dual algorithms
- An adaptive gradient method with energy and momentum
Cites work
- scientific article; zbMATH DE number 3850830 (Why is no real title available?)
- scientific article; zbMATH DE number 3790208 (Why is no real title available?)
- scientific article; zbMATH DE number 1993614 (Why is no real title available?)
- scientific article; zbMATH DE number 1487987 (Why is no real title available?)
- A differential equation for modeling Nesterov's accelerated gradient method: theory and insights
- A new approach to computing maximum flows using electrical flows
- A second-order differential system with Hessian-driven damping; application to non-elastic shock laws
- A variational perspective on accelerated methods in optimization
- Accelerated extra-gradient descent: a novel accelerated first-order method
- Accelerated gradient methods for nonconvex nonlinear and stochastic programming
- Accelerated primal-dual gradient descent with linesearch for convex, nonconvex, and nonsmooth optimization problems
- An almost-linear-time algorithm for approximate max flow in undirected graphs, and its multicommodity generalizations
- An optimal first order method based on optimal quadratic averaging
- Area-convexity, \(\ell_\infty\) regularization, and undirected multicommodity flow
- Behavior of accelerated gradient methods near critical points of nonconvex functions
- Catalyst acceleration for first-order convex optimization: from theory to practice
- Characterizations of Łojasiewicz inequalities: Subgradient flows, talweg, convexity
- Conformal symplectic and relativistic optimization
- Discrete processes and their continuous limits
- First-order optimization algorithms via inertial systems with Hessian driven damping
- Generalized uniformly optimal methods for nonlinear programming
- Generalizing the optimized gradient method for smooth convex minimization
- Inertial game dynamics and applications to constrained optimization
- Lectures on convex optimization
- Linear coupling: an ultimate unification of gradient and mirror descent
- Lower bounds for finding stationary points I
- Lower bounds for parallel and randomized convex optimization
- New Proximal Point Algorithms for Convex Minimization
- On damped second-order gradient systems
- On the convergence of the iterates of the ``fast iterative shrinkage/thresholding algorithm
- On the long time behavior of second order differential equations with asymptotically small dissipation
- Optimal methods of smooth convex minimization
- Optimizing the efficiency of first-order methods for decreasing the gradient of smooth convex functions
- Proximally guided stochastic subgradient method for nonsmooth, nonconvex problems
- Rate of convergence of the Nesterov accelerated gradient method in the subcritical case α ≤ 3
- Some methods of speeding up the convergence of iteration methods
- THE HEAVY BALL WITH FRICTION METHOD, I. THE CONTINUOUS DYNAMICAL SYSTEM: GLOBAL EXPLORATION OF THE LOCAL MINIMA OF A REAL-VALUED FUNCTION BY ASYMPTOTIC ANALYSIS OF A DISSIPATIVE DYNAMICAL SYSTEM
- The approximate duality gap technique: a unified theory of first-order methods
- Universal method for stochastic composite optimization problems
Cited in
(12)- A Systematic Approach to Lyapunov Analyses of Continuous-Time Models in Convex Optimization
- Implicit Regularization and Momentum Algorithms in Nonlinearly Parameterized Adaptive Control and Prediction
- Asymptotic analysis of a structure-preserving integrator for damped Hamiltonian systems
- Fast symplectic integrator for Nesterov-type acceleration method
- Iteration complexity of fixed-step methods by Nesterov and Polyak for convex quadratic functions
- A variational perspective on accelerated methods in optimization
- Momentum and Hamiltonian operators in generalized coordinates
- Recent Theoretical Advances in Non-Convex Optimization
- On the convergence analysis of aggregated heavy-ball method
- Bregman dynamics, contact transformations and convex optimization
- Potential Function-Based Framework for Minimizing Gradients in Convex and Min-Max Optimization
- Convergence of the Momentum Method for Semialgebraic Functions with Locally Lipschitz Gradients
This page was built for publication: Generalized momentum-based methods: a Hamiltonian perspective
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5857293)
