A variational formulation of accelerated optimization on Riemannian manifolds

DOI10.1137/21M1395648zbMATH Open1504.37104arXiv2101.06552OpenAlexW3124961975WikidataQ115246868 ScholiaQ115246868MaRDI QIDQ5863993FDOQ5863993

Authors: Valentin Duruisseaux, Melvin Leok 

Publication date: 3 June 2022

Published in: SIAM Journal on Mathematics of Data Science (Search for Journal in Brave)

Abstract: It was shown recently by Su et al. (2016) that Nesterov's accelerated gradient method for minimizing a smooth convex function $f$ can be thought of as the time discretization of a second-order ODE, and that $f(x(t))$ converges to its optimal value at a rate of $mathcalO(1/t^2)$ along any trajectory $x(t)$ of this ODE. A variational formulation was introduced in Wibisono et al. (2016) which allowed for accelerated convergence at a rate of $mathcalO(1/t^p)$, for arbitrary $p>0$, in normed vector spaces. This framework was exploited in Duruisseaux et al. (2021) to design efficient explicit algorithms for symplectic accelerated optimization. In Alimisis et al. (2020), a second-order ODE was proposed as the continuous-time limit of a Riemannian accelerated algorithm, and it was shown that the objective function $f(x(t))$ converges to its optimal value at a rate of $mathcalO(1/t^2)$ along solutions of this ODE. In this paper, we show that on Riemannian manifolds, the convergence rate of $f(x(t))$ to its optimal value can also be accelerated to an arbitrary convergence rate $mathcalO(1/t^p)$, by considering a family of time-dependent Bregman Lagrangian and Hamiltonian systems on Riemannian manifolds. This generalizes the results of Wibisono et al. (2016) to Riemannian manifolds and also provides a variational framework for accelerated optimization on Riemannian manifolds. An approach based on the time-invariance property of the family of Bregman Lagrangians and Hamiltonians was used to construct very efficient optimization algorithms in Duruisseaux et al. (2021), and we establish a similar time-invariance property in the Riemannian setting. One expects that a geometric numerical integrator that is time-adaptive, symplectic, and Riemannian manifold preserving will yield a class of promising optimization algorithms on manifolds.

Full work available at URL: https://arxiv.org/abs/2101.06552

Recommendations

• Accelerated optimization on Riemannian manifolds via discrete constrained variational integrators
• A variational perspective on accelerated methods in optimization
• Accelerated Optimization in the PDE Framework: Formulations for the Manifold of Diffeomorphisms
• Riemannian proximal gradient methods
• scientific article; zbMATH DE number 1146390

zbMATH Keywords

Riemannian optimizationaccelerated optimizationsymplectic optimizationNesterov accelerated gradient

Mathematics Subject Classification ID

Numerical optimization and variational techniques (65K10) Dynamical systems in optimization and economics (37N40) Numerical methods for Hamiltonian systems including symplectic integrators (65P10) Canonical and symplectic transformations for problems in Hamiltonian and Lagrangian mechanics (70H15)

Cites Work

• Introductory lectures on convex optimization. A basic course.
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Discrete mechanics and variational integrators
• Title not available (Why is that?)
• Geometric Numerical Integration
• Title not available (Why is that?)
• Title not available (Why is that?)
• A differential equation for modeling Nesterov's accelerated gradient method: theory and insights
• Accelerating the cubic regularization of Newton's method on convex problems
• Variational and geometric structures of discrete Dirac mechanics
• Introduction to Riemannian Manifolds
• Riemannian geometry and geometric analysis
• A variational perspective on accelerated methods in optimization
• Adaptive Hamiltonian variational integrators and applications to symplectic accelerated optimization
• Accelerated optimization on Riemannian manifolds via discrete constrained variational integrators

Cited In (8)

• Practical perspectives on symplectic accelerated optimization
• Fast gradient method for low-rank matrix estimation
• Accelerated optimization on Riemannian manifolds via discrete constrained variational integrators
• Accelerated Optimization in the PDE Framework: Formulations for the Manifold of Diffeomorphisms
• Bregman dynamics, contact transformations and convex optimization
• Time-adaptive Lagrangian variational integrators for accelerated optimization
• A variational perspective on accelerated methods in optimization
• High-order symplectic Lie group methods on \(SO(n)\) using the polar decomposition

This page was built for publication: A variational formulation of accelerated optimization on Riemannian manifolds

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5863993)