Riemannian Stochastic Variance Reduced Gradient Algorithm with Retraction and Vector Transport
From MaRDI portal
Abstract: In recent years, stochastic variance reduction algorithms have attracted considerable attention for minimizing the average of a large but finite number of loss functions. This paper proposes a novel Riemannian extension of the Euclidean stochastic variance reduced gradient (R-SVRG) algorithm to a manifold search space. The key challenges of averaging, adding, and subtracting multiple gradients are addressed with retraction and vector transport. For the proposed algorithm, we present a global convergence analysis with a decaying step size as well as a local convergence rate analysis with a fixed step size under some natural assumptions. In addition, the proposed algorithm is applied to the computation problem of the Riemannian centroid on the symmetric positive definite (SPD) manifold as well as the principal component analysis and low-rank matrix completion problems on the Grassmann manifold. The results show that the proposed algorithm outperforms the standard Riemannian stochastic gradient descent algorithm in each case.
Recommendations
- Stochastic Gradient Descent on Riemannian Manifolds
- Riemannian proximal gradient methods
- Riemannian conjugate gradient methods with inverse retraction
- Riemannian stochastic variance-reduced cubic regularized Newton method for submanifold optimization
- Sufficient descent Riemannian conjugate gradient methods
- An inexact Riemannian proximal gradient method
- Riemannian stochastic fixed point optimization algorithm
- Recent Advances in Stochastic Riemannian Optimization
- Riemannian gradient methods for stochastic composition problems
- Stochastic variance reduced gradient methods using a trust-region-like scheme
Cites work
- A Broyden class of quasi-Newton methods for Riemannian optimization
- A proximal stochastic gradient method with progressive variance reduction
- A Riemannian framework for tensor computing
- A Riemannian symmetric rank-one trust-region method
- A survey and comparison of contemporary algorithms for computing the matrix geometric mean
- Eigentaste: A constant time collaborative filtering algorithm
- scientific article; zbMATH DE number 5223994 (Why is no real title available?)
- Incremental majorization-minimization optimization with application to large-scale machine learning
- Low-rank tensor completion by Riemannian optimization
- Manopt, a Matlab toolbox for optimization on manifolds
- Quasi-Martingales
- Riemannian preconditioning
- Stochastic dual coordinate ascent methods for regularized loss minimization
- Stochastic Gradient Descent on Riemannian Manifolds
Cited in
(32)- Riemannian conjugate gradient methods with inverse retraction
- Sequential optimality conditions for nonlinear optimization on Riemannian manifolds and a globally convergent augmented Lagrangian method
- Measure estimation on manifolds: an optimal transport approach
- Riemannian stochastic fixed point optimization algorithm
- A brief introduction to manifold optimization
- Cayley-transform-based gradient and conjugate gradient algorithms on Grassmann manifolds
- A Riemannian gossip approach to subspace learning on Grassmann manifold
- An efficient damped Newton-type algorithm with globalization strategy on Riemannian manifolds
- Riemannian stochastic variance-reduced cubic regularized Newton method for submanifold optimization
- Geometric Methods on Low-Rank Matrix and Tensor Manifolds
- Recent Advances in Stochastic Riemannian Optimization
- Some limit properties of Markov chains induced by recursive stochastic algorithms
- Projection-free nonconvex stochastic optimization on Riemannian manifolds
- Riemannian conjugate gradient methods: general framework and specific algorithms with convergence analyses
- Sequential quadratic optimization for nonlinear optimization problems on Riemannian manifolds
- Optimizing oblique projections for nonlinear systems using trajectories
- Stochastic Gradient Descent on Riemannian Manifolds
- Riemannian smoothing gradient type algorithms for nonsmooth optimization problem on compact Riemannian submanifold embedded in Euclidean space
- Sion’s Minimax Theorem in Geodesic Metric Spaces and a Riemannian Extragradient Algorithm
- Riemannian Hamiltonian Methods for Min-Max Optimization on Manifolds
- Riemannian optimization on unit sphere with \(p\)-norm and its applications
- A variance-reduced stochastic gradient tracking algorithm for decentralized optimization with orthogonality constraints
- Riemannian Natural Gradient Methods
- Riemannian gradient methods for stochastic composition problems
- Practical gradient and conjugate gradient methods on flag manifolds
- Convergence of variational Monte Carlo simulation and scale-invariant pre-training
- Zeroth-order Riemannian averaging stochastic approximation algorithms
- Stochastic augmented Lagrangian method in Riemannian shape manifolds
- Convergence of Riemannian stochastic gradient descent on Hadamard manifold
- Accelerated gradient methods on the Grassmann and Stiefel manifolds
- Global Riemannian acceleration in hyperbolic and spherical spaces
- Optimization, complexity and invariant theory (invited talk)
This page was built for publication: Riemannian Stochastic Variance Reduced Gradient Algorithm with Retraction and Vector Transport
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5231672)
