Practical perspectives on symplectic accelerated optimization
From MaRDI portal
accelerated optimizationBregman dynamicsgeometric numerical integrationHamiltonian dynamicsLagrangian dynamicssymplectic integratorsvariational integrators
Discretization methods and integrators (symplectic, variational, geometric, etc.) for dynamical systems (37M15) Dynamical systems in optimization and economics (37N40) Numerical optimization and variational techniques (65K10) Numerical methods for Hamiltonian systems including symplectic integrators (65P10) Canonical and symplectic transformations for problems in Hamiltonian and Lagrangian mechanics (70H15)
Abstract: Geometric numerical integration has recently been exploited to design symplectic accelerated optimization algorithms by simulating the Lagrangian and Hamiltonian systems from the variational framework introduced in Wibisono et al. In this paper, we discuss practical considerations which can significantly boost the computational performance of these optimization algorithms, and considerably simplify the tuning process. In particular, we investigate how momentum restarting schemes ameliorate computational efficiency and robustness by reducing the undesirable effect of oscillations, and ease the tuning process by making time-adaptivity superfluous. We also discuss how temporal looping helps avoiding instability issues caused by numerical precision, without harming the computational efficiency of the algorithms. Finally, we compare the efficiency and robustness of different geometric integration techniques, and study the effects of the different parameters in the algorithms to inform and simplify tuning in practice. From this paper emerge symplectic accelerated optimization algorithms whose computational efficiency, stability and robustness have been improved, and which are now much simpler to use and tune for practical applications.
Recommendations
- scientific article; zbMATH DE number 1193157
- On dissipative symplectic integration with applications to gradient-based optimization
- Symplectic Pontryagin approximations for optimal design
- Symplectic geometry of constrained optimization
- Adaptive Hamiltonian variational integrators and applications to symplectic accelerated optimization
- DYNAMICAL, SYMPLECTIC AND STOCHASTIC PERSPECTIVES ON GRADIENT-BASED OPTIMIZATION
- scientific article; zbMATH DE number 1069101
- Conformal symplectic and relativistic optimization
- Optimization on the real symplectic group
- An improved symplectic method for solving optimal control problems
Cites work
- A concise introduction to geometric numerical integration
- A differential equation for modeling Nesterov's accelerated gradient method: theory and insights
- A second-order gradient-like dissipative dynamical system with Hessian-driven damping. Application to optimization and mechanics.
- A simple nearly optimal restart scheme for speeding up first-order methods
- A Technique for the Numerical Solution of Certain Integral Equations of the First Kind
- A variational formulation of accelerated optimization on Riemannian manifolds
- A variational perspective on accelerated methods in optimization
- Accelerated optimization on Riemannian manifolds via discrete constrained variational integrators
- Accelerating the cubic regularization of Newton's method on convex problems
- Adaptive Hamiltonian variational integrators and applications to symplectic accelerated optimization
- Adaptive restart for accelerated gradient schemes
- Adaptive restart of accelerated gradient methods under local quadratic growth condition
- Adaptive restart of the optimized gradient method for convex optimization
- Asymptotic behavior of gradient-like dynamical systems involving inertia and multiscale aspects
- Averaging methods in nonlinear dynamical systems
- Combining fast inertial dynamics for convex optimization with Tikhonov regularization
- Computing Highly Oscillatory Integrals
- Convergence of iterates for first-order optimization algorithms with inertia and Hessian driven damping
- Convex optimization theory.
- Discrete Hamiltonian variational integrators
- Discrete mechanics and variational integrators
- Discrete variational Hamiltonian mechanics
- DYNAMICAL, SYMPLECTIC AND STOCHASTIC PERSPECTIVES ON GRADIENT-BASED OPTIMIZATION
- Fast convex optimization via inertial dynamics combining viscous and Hessian-driven damping with time rescaling
- Fast integration of rapidly oscillatory functions
- First-order optimization algorithms via inertial systems with Hessian driven damping
- General techniques for constructing variational integrators
- Geometric methods and optimization problems
- Geometric Numerical Integration
- Geometric numerical integration illustrated by the Störmer–Verlet method
- Gradient-based algorithms with applications to signal-recovery problems
- scientific article; zbMATH DE number 3850830 (Why is no real title available?)
- scientific article; zbMATH DE number 3790208 (Why is no real title available?)
- scientific article; zbMATH DE number 3551792 (Why is no real title available?)
- scientific article; zbMATH DE number 1012640 (Why is no real title available?)
- scientific article; zbMATH DE number 1746287 (Why is no real title available?)
- scientific article; zbMATH DE number 940595 (Why is no real title available?)
- scientific article; zbMATH DE number 2107836 (Why is no real title available?)
- scientific article; zbMATH DE number 845714 (Why is no real title available?)
- scientific article; zbMATH DE number 3227378 (Why is no real title available?)
- Introductory lectures on convex optimization. A basic course.
- Lagrangian and Hamiltonian Taylor variational integrators
- Lie Groups, Lie Algebras, and Representations
- On an asymptotically autonomous system with Tikhonov type regularizing term
- On restart procedures for the conjugate gradient method
- On the Hamiltonian interpolation of near-to-the-identity symplectic mappings with application to symplectic integration algorithms
- Procedures for Computing One- and Two-Dimensional Integrals of Functions with Rapid Irregular Oscillations
- Prolongation-collocation variational integrators
- Properties of Hamiltonian variational integrators
- Restart procedures for the conjugate gradient method
- Restarting the accelerated coordinate descent method with a rough strong convexity estimate
- Sharpness, restart, and acceleration
- Simulating Hamiltonian Dynamics
- Symplectic integrators for long-term integrations in celestial mechanics
- The Development of Variable-Step Symplectic Integrators, with Application to the Two-Body Problem
- Tikhonov regularization of a perturbed heavy ball system with vanishing damping
- Time transformations in the extended phase-space
- Time-adaptive Lagrangian variational integrators for accelerated optimization
- Variable time step integration with symplectic methods
- Variational integrators for almost-integrable systems
Cited in
(10)- On dissipative symplectic integration with applications to gradient-based optimization
- Symplectic Pontryagin approximations for optimal design
- Conformal symplectic and relativistic optimization
- Time-adaptive Lagrangian variational integrators for accelerated optimization
- The application of Goeken-Johnson's Runge-Kutta methods in unconstrained convex optimization
- Variational principles for Hamiltonian systems
- Application of the modified Runge-Kutta method to the construction of the descent method for solving boundary value problems
- Application of Runge-Kutta method with non-standard finite difference for the construction of preconditioned gradient methods
- Gradient methods based on non-standard Lagrange-Burmann Runge-Kutta method
- On the application of explicit Runge-Kutta methods to the construction of stochastic gradient descent methods for convex optimization
This page was built for publication: Practical perspectives on symplectic accelerated optimization
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6087057)
