Friction-adaptive descent: a family of dynamics-based optimization methods

DOI10.3934/JCD.2023007arXiv2306.06738OpenAlexW4386080447MaRDI QIDQ6087370FDOQ6087370

Authors: Aikaterini Karoni, Benedict Leimkuhler, Gabriel Stoltz 

Publication date: 15 November 2023

Published in: Journal of Computational Dynamics (Search for Journal in Brave)

Abstract: We describe a family of descent algorithms which generalizes common existing schemes used in applications such as neural network training and more broadly for optimization of smooth functions--potentially for global optimization, or as a local optimization method to be deployed within global optimization schemes like basin hopping. By introducing an auxiliary degree of freedom we create a dynamical system with improved stability, reducing oscillatory modes and accelerating convergence to minima. The resulting algorithms are simple to implement and control, and convergence can be shown directly by Lyapunov's second method. Although this framework, which we refer to as friction-adaptive descent (FAD), is fairly general, we focus most of our attention here on a specific variant: kinetic energy stabilization (which can be viewed as a zero-temperature Nos'e--Hoover scheme but with added dissipation in both physical and auxiliary variables), termed KFAD (kinetic FAD). To illustrate the flexibility of the FAD framework we consider several other methods. in certain asymptotic limits, these methods can be viewed as introducing cubic damping in various forms; they can be more efficient than linearly dissipated Hamiltonian dynamics in common optimization settings. We present details of the numerical methods and show convergence for both the continuous and discretized dynamics in the convex setting by constructing Lyapunov functions. The methods are tested using a toy model (the Rosenbrock function). We also demonstrate the methods for structural optimization for atomic clusters in Lennard--Jones and Morse potentials. The experiments show the relative efficiency and robustness of FAD in comparison to linearly dissipated Hamiltonian dynamics.

Full work available at URL: https://arxiv.org/abs/2306.06738

Recommendations

• An adaptive gradient method with energy and momentum
• First order inertial optimization algorithms with threshold effects associated with dry friction
• Steepest descent with momentum for quadratic functions is a version of the conjugate gradient method
• Neurodynamical optimization

zbMATH Keywords

Hamiltonian systemgradient descentcubic dampingconvex and nonconvex optimizationNosé-Hoover dynamicsadaptive LangevinPolyak heavy ball method

Mathematics Subject Classification ID

Numerical optimization and variational techniques (65K10) Simulation of dynamical systems (37M05) Discretization methods and integrators (symplectic, variational, geometric, etc.) for dynamical systems (37M15) Computational molecular dynamics in statistical mechanics (82M37) Computational methods for attractors of dynamical systems (37M22)

Cites Work

• Title not available (Why is that?)
• Title not available (Why is that?)
• Introductory lectures on convex optimization. A basic course.
• Deep learning
• Title not available (Why is that?)
• Splitting methods
• Ergodicity for SDEs and approximations: locally Lipschitz vector fields and degenerate noise.
• A differential equation for modeling Nesterov's accelerated gradient method: theory and insights
• Some methods of speeding up the convergence of iteration methods
• Langevin dynamics with general kinetic energies
• Rational construction of stochastic numerical methods for molecular sampling
• Title not available (Why is that?)
• A gentle stochastic thermostat for molecular dynamics
• Conformal Hamiltonian systems
• Second order conformal symplectic schemes for damped Hamiltonian systems
• A Projective Thermostatting Dynamics Technique
• Hypocoercivity properties of adaptive Langevin dynamics
• Exponential relaxation of the Nosé-Hoover thermostat under Brownian heating
• A Systematic Approach to Lyapunov Analyses of Continuous-Time Models in Convex Optimization
• A general system of differential equations to model first-order adaptive algorithms

This page was built for publication: Friction-adaptive descent: a family of dynamics-based optimization methods

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6087370)