Invariance properties of the natural gradient in overparametrised systems

DOI10.1007/S41884-022-00067-9MaRDI QIDQ6115103zbMATH OpenFDO

Authors

Publication date 12 July 2023

Published in Information Geometry (Search for Journal in Brave)

Copyright license Creative Commons Attribution 4.0 International

Full work available at URL https://arxiv.org/abs/2206.15273

zbMATH Keywords

deep learninginformation geometrynatural gradientRiemannian metric

Mathematics Subject Classification ID

Sufficiency and information (62Bxx) Artificial intelligence (68Txx)

Abstract: The natural gradient field is a vector field that lives on a model equipped with a distinguished Riemannian metric, e.g. the Fisher-Rao metric, and represents the direction of steepest ascent of an objective function on the model with respect to this metric. In practice, one tries to obtain the corresponding direction on the parameter space by multiplying the ordinary gradient by the inverse of the Gram matrix associated with the metric. We refer to this vector on the parameter space as the natural parameter gradient. In this paper we study when the pushforward of the natural parameter gradient is equal to the natural gradient. Furthermore we investigate the invariance properties of the natural parameter gradient. Both questions are addressed in an overparametrised setting.

Recommendations

• Parametrisation independence of the natural gradient in overparametrised systems
• scientific article; zbMATH DE number 1728685
• scientific article; zbMATH DE number 7306852
• Loss landscapes and optimization in over-parameterized non-linear systems and neural networks
• Stability analysis of natural gradient learning rules in overdetermined ICA
• Dynamics of learning in MLP: natural gradient and singularity revisited
• Convergence properties of the gradient method under conditions of variable-level interference
• scientific article; zbMATH DE number 1728687
• of overparametrization in non-linear system identification and neural networks

Cites work

• scientific article; zbMATH DE number 7306852 (Why is no real title available?)
• Natural gradient via optimal transport
• On the locality of the natural gradient for learning in deep Bayesian networks
• Parametrisation independence of the natural gradient in overparametrised systems
• Riemannian metrics for neural networks. I: Feedforward networks
• Stochastic approximation methods for constrained and unconstrained systems
• Wasserstein Riemannian geometry of Gaussian densities

Cited in

(3)

• Geometry and convergence of natural policy gradient methods
• Fisher-Rao gradient flows of linear programs and state-action natural policy gradients
• On the natural gradient of the evidence lower bound

This page was built for publication: Invariance properties of the natural gradient in overparametrised systems

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6115103)