Riemannian Natural Gradient Methods
From MaRDI portal
Abstract: This paper studies large-scale optimization problems on Riemannian manifolds whose objective function is a finite sum of negative log-probability losses. Such problems arise in various machine learning and signal processing applications. By introducing the notion of Fisher information matrix in the manifold setting, we propose a novel Riemannian natural gradient method, which can be viewed as a natural extension of the natural gradient method from the Euclidean setting to the manifold setting. We establish the almost-sure global convergence of our proposed method under standard assumptions. Moreover, we show that if the loss function satisfies certain convexity and smoothness conditions and the input-output map satisfies a Riemannian Jacobian stability condition, then our proposed method enjoys a local linear -- or, under the Lipschitz continuity of the Riemannian Jacobian of the input-output map, even quadratic -- rate of convergence. We then prove that the Riemannian Jacobian stability condition will be satisfied by a two-layer fully connected neural network with batch normalization with high probability, provided that the width of the network is sufficiently large. This demonstrates the practical relevance of our convergence rate result. Numerical experiments on applications arising from machine learning demonstrate the advantages of the proposed method over state-of-the-art ones.
Recommendations
- Riemannian gradient methods for stochastic composition problems
- Convergence of Riemannian stochastic gradient descent on Hadamard manifold
- Recent Advances in Stochastic Riemannian Optimization
- Riemannian proximal gradient methods
- Faster Riemannian Newton-type optimization by subsampling and cubic regularization
Cites work
- A brief introduction to manifold optimization
- A Riemannian gossip approach to subspace learning on Grassmann manifold
- A Stochastic Approximation Method
- A stochastic extra-step quasi-Newton method for nonsmooth nonconvex optimization
- A stochastic quasi-Newton method for large-scale optimization
- Adaptive quadratically regularized Newton method for Riemannian optimization
- Adaptive subgradient methods for online learning and stochastic optimization
- An Extrinsic Look at the Riemannian Hessian
- An Introduction to Optimization on Smooth Manifolds
- Covariance, subspace, and intrinsic Crame/spl acute/r-Rao bounds
- Deep learning
- Differentiation Under the Integral Sign
- Efficient Natural Gradient Descent Methods for Large-Scale PDE-Based Optimization Problems
- Fast Curvature Matrix-Vector Products for Second-Order Gradient Descent
- scientific article; zbMATH DE number 5957307 (Why is no real title available?)
- scientific article; zbMATH DE number 1871413 (Why is no real title available?)
- scientific article; zbMATH DE number 775283 (Why is no real title available?)
- scientific article; zbMATH DE number 7306852 (Why is no real title available?)
- scientific article; zbMATH DE number 5223994 (Why is no real title available?)
- scientific article; zbMATH DE number 5060482 (Why is no real title available?)
- Information-geometric optimization algorithms: a unifying picture via invariance principles
- Low-rank matrix completion via preconditioned optimization on the Grassmann manifold
- Newton Sketch: A Near Linear-Time Optimization Algorithm with Linear-Quadratic Convergence
- Optimization methods on Riemannian manifolds and their application to shape space
- Projection-like retractions on matrix manifolds
- Riemannian optimization and its applications
- Riemannian Stochastic Variance Reduced Gradient Algorithm with Retraction and Vector Transport
- Riemannian stochastic variance-reduced cubic regularized Newton method for submanifold optimization
- Sketch-based empirical natural gradient methods for deep learning
- Stochastic Gradient Descent on Riemannian Manifolds
- Stochastic optimization using a trust-region method and random models
- Stochastic trust-region methods with trust-region radius depending on probabilistic models
- Sub-sampled Newton methods
Cited in
(5)- Understanding approximate Fisher information for fast convergence of natural gradient descent in wide neural networks*
- Riemannian gradient methods for stochastic composition problems
- Convergence of hyperbolic neural networks under Riemannian stochastic gradient descent
- Convergence of Riemannian stochastic gradient descent on Hadamard manifold
- Decentralized projected Riemannian gradient method for smooth optimization on compact submanifolds embedded in the Euclidean space
This page was built for publication: Riemannian Natural Gradient Methods
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6189169)
