A variational perspective on accelerated methods in optimization
From MaRDI portal
Abstract: Accelerated gradient methods play a central role in optimization, achieving optimal rates in many settings. While many generalizations and extensions of Nesterov's original acceleration method have been proposed, it is not yet clear what is the natural scope of the acceleration concept. In this paper, we study accelerated methods from a continuous-time perspective. We show that there is a Lagrangian functional that we call the emph{Bregman Lagrangian} which generates a large class of accelerated methods in continuous time, including (but not limited to) accelerated gradient descent, its non-Euclidean extension, and accelerated higher-order gradient methods. We show that the continuous-time limit of all of these methods correspond to traveling the same curve in spacetime at different speeds. From this perspective, Nesterov's technique and many of its generalizations can be viewed as a systematic way to go from the continuous-time curves generated by the Bregman Lagrangian to a family of discrete-time accelerated algorithms.
Recommendations
- Multiscale analysis of accelerated gradient methods
- A variational formulation of accelerated optimization on Riemannian manifolds
- Understanding the acceleration phenomenon via high-resolution differential equations
- A differential equation for modeling Nesterov's accelerated gradient method: theory and insights
- Generalized momentum-based methods: a Hamiltonian perspective
Cited in
(only showing first 100 items - show all)- From differential equation solvers to accelerated first-order methods for convex optimization
- Stochastic modified equations and dynamics of stochastic gradient algorithms. I: Mathematical foundations
- Analyze accelerated mirror descent via high-resolution ODEs
- Near-optimal tensor methods for minimizing the gradient norm of convex functions and accelerated primal–dual tensor methods
- Decentralized concurrent learning with coordinated momentum and restart
- Projected dynamical systems on irregular, non-Euclidean domains for nonlinear optimization
- Accelerated primal-dual methods for strongly convex objective functions in continuous and discrete time
- Finite and fixed-time feedback-based continuous-time optimization
- The connections between Lyapunov functions for some optimization algorithms and differential equations
- Essential convergence rate of ordinary differential equations appearing in optimization
- Revisiting the ODE method for recursive algorithms: fast convergence using quasi stochastic approximation
- Understanding the acceleration phenomenon via high-resolution differential equations
- A gradient-based optimization method using the Koopman operator
- A unified differential equation solver approach for separable convex optimization: splitting, acceleration and nonergodic rate
- Variational principles for Hamiltonian systems
- Energy-stable swarm-based inertial algorithms for optimization
- Generalized momentum-based methods: a Hamiltonian perspective
- Fast symplectic integrator for Nesterov-type acceleration method
- Tikhonov regularization of second-order plus first-order primal-dual dynamical systems for separable convex optimization
- Convergence rate bounds for the mirror descent method: IQCs, Popov criterion and Bregman divergence
- A stochastic iteratively regularized Gauss-Newton method
- Robust hybrid zero-order optimization algorithms with acceleration via averaging in time
- How does momentum benefit deep neural networks architecture design? A few case studies
- Exponential convergence of distributed primal-dual convex optimization algorithm without strong convexity
- A geometric integration approach to smooth optimization: foundations of the discrete gradient method
- Fast convergence rates and trajectory convergence of a Tikhonov regularized inertial primal-dual dynamical system with time scaling and vanishing damping
- Inertial accelerated augmented Lagrangian algorithms with scaling coefficients to solve exactly and inexactly linearly constrained convex optimization problems
- Perseus: a simple and optimal high-order method for variational inequalities
- Accelerations for global optimization covering methods using second derivatives
- Unified acceleration of high-order algorithms under general Hölder continuity
- Accelerating Proximal Markov Chain Monte Carlo by Using an Explicit Stabilized Method
- Inertial primal-dual dynamics with damping and scaling for linearly constrained convex optimization problems
- Contractivity of Runge-Kutta methods for convex gradient systems
- Bregman Itoh-Abe methods for sparse optimisation
- Adaptive Hamiltonian variational integrators and applications to symplectic accelerated optimization
- A variational formulation of accelerated optimization on Riemannian manifolds
- No-regret algorithms in on-line learning, games and convex optimization
- Accelerating optimization over the space of probability measures
- High-order symplectic Lie group methods on \(SO(n)\) using the polar decomposition
- Finding extremals of Lagrangian actions
- Deterministic neural networks optimization from a continuous and energy point of view
- A general mixed-order primal-dual dynamical system with Tikhonov regularization
- Finding geodesics joining given points
- Convergence rates of inertial primal-dual dynamical methods for separable convex optimization problems
- Preconditioned accelerated gradient descent methods for locally Lipschitz smooth objectives with applications to the solution of nonlinear PDEs
- An LQR-Lagrange algorithm generated by interdisciplinary integration with optimal control and optimization
- A differential equation for modeling Nesterov's accelerated gradient method: theory and insights
- No-regret dynamics in the Fenchel game: a unified framework for algorithmic convex optimization
- Discrete processes and their continuous limits
- Fast primal-dual algorithm via dynamical system for a linearly constrained convex optimization problem
- Online optimization of switched LTI systems using continuous-time and hybrid accelerated gradient flows
- New methods for parametric optimization via differential equations
- Online accelerated data-driven learning for optimal feedback control of discrete-time partially uncertain systems
- A fixed-time stable forward-backward dynamical system for solving generalized monotone inclusions
- Dynamical Systems Theory and Algorithms for NP-hard Problems
- An optimal high-order tensor method for convex optimization
- Triggered gradient tracking for asynchronous distributed optimization
- Inducing strong convergence of trajectories in dynamical systems associated to monotone inclusions with composite structure
- Explicit stabilised gradient descent for faster strongly convex optimisation
- Conformal mirror descent with logarithmic divergences
- Potential Function-Based Framework for Minimizing Gradients in Convex and Min-Max Optimization
- scientific article; zbMATH DE number 1268598 (Why is no real title available?)
- Linear convergence of forward-backward accelerated algorithms without knowledge of the modulus of strong convexity
- A fast primal-dual algorithm via dynamical system with variable mass for linearly constrained convex optimization
- On the connections between optimization algorithms, Lyapunov functions, and differential equations: theory and insights
- PDE acceleration: a convergence rate analysis and applications to obstacle problems
- Conformal symplectic and relativistic optimization
- Stochastic Methods for Composite and Weakly Convex Optimization Problems
- Mini-workshop: High-dimensional control problems and mean-field equations with applications in machine learning. Abstracts from the mini-workshop held December 8--13, 2024
- A general system of differential equations to model first-order adaptive algorithms
- Proximal gradient flow and Douglas-Rachford splitting dynamics: global exponential stability via integral quadratic constraints
- A new dynamical system with self-adaptive dynamical stepsize for pseudomonotone mixed variational inequalities
- Practical perspectives on symplectic accelerated optimization
- On dissipative symplectic integration with applications to gradient-based optimization
- First-order methods for convex optimization
- A primal-dual flow for affine constrained convex optimization
- Novel projection neurodynamic approaches for constrained convex optimization
- Implicit Regularization and Momentum Algorithms in Nonlinearly Parameterized Adaptive Control and Prediction
- Differential equations in data analysis
- Selection dynamics for deep neural networks
- A Recursively Recurrent Neural Network (R2N2) Architecture for Learning Iterative Algorithms
- Entropic mean-field min-max problems via best response flow
- A Systematic Approach to Lyapunov Analyses of Continuous-Time Models in Convex Optimization
- Accelerated variational PDEs for efficient solution of regularized inversion problems
- Multiscale analysis of accelerated gradient methods
- Accelerated Forward-Backward Optimization using Deep Learning
- Fast gradient method for low-rank matrix estimation
- A Continuous-Time Analysis of Distributed Stochastic Gradient
- The regularized submodular maximization via the Lyapunov method
- On the convergence of gradient-like flows with noisy gradient input
- Accelerated forward-backward and Douglas-Rachford splitting dynamics
- Continuous relaxations for the traveling salesman problem
- Accelerated optimization on Riemannian manifolds via discrete constrained variational integrators
- scientific article; zbMATH DE number 7370630 (Why is no real title available?)
- Accelerated Optimization in the PDE Framework: Formulations for the Manifold of Diffeomorphisms
- A continuous-time perspective on global acceleration for monotone equation problems
- Bregman dynamics, contact transformations and convex optimization
- Convergence rates of mixed primal-dual dynamical systems with Hessian driven damping
- Accelerated Optimization in the PDE Framework Formulations for the Active Contour Case
- Accelerated first-order primal-dual proximal methods for linearly constrained composite convex programming
This page was built for publication: A variational perspective on accelerated methods in optimization
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q4646231)
