On the locality of the natural gradient for learning in deep Bayesian networks

DOI10.1007/S41884-020-00038-YMaRDI QIDQ6115102zbMATH OpenOpenAlexFDO

Authors Nihat Ay

Publication date 12 July 2023

Published in Information Geometry (Search for Journal in Brave)

Copyright license Creative Commons Attribution 4.0 International

Full work available at URL https://arxiv.org/abs/2005.10791

zbMATH Keywords

deep learningFisher-Rao metricHelmholtz machinesnatural gradientwake-sleep algorithm

Mathematics Subject Classification ID

Basic linear algebra (15Axx) Local differential geometry (53Bxx) Artificial intelligence (68Txx)

Abstract: We study the natural gradient method for learning in deep Bayesian networks, including neural networks. There are two natural geometries associated with such learning systems consisting of visible and hidden units. One geometry is related to the full system, the other one to the visible sub-system. These two geometries imply different natural gradients. In a first step, we demonstrate a great simplification of the natural gradient with respect to the first geometry, due to locality properties of the Fisher information matrix. This simplification does not directly translate to a corresponding simplification with respect to the second geometry. We develop the theory for studying the relation between the two versions of the natural gradient and outline a method for the simplification of the natural gradient with respect to the second geometry based on the first one. This method suggests to incorporate a recognition model as an auxiliary model for the efficient application of the natural gradient method in deep networks.

Cites work

• Algebraic Geometry and Statistical Learning Theory
• Connectionist learning of belief networks
• Deep learning
• Expressions for the Moore-Penrose inverse of block matrices involving the Schur complement
• Gradient systems in view of information geometry
• scientific article; zbMATH DE number 3761167 (Why is no real title available?)
• scientific article; zbMATH DE number 1134987 (Why is no real title available?)
• scientific article; zbMATH DE number 1560711 (Why is no real title available?)
• scientific article; zbMATH DE number 7306852 (Why is no real title available?)
• Information geometry
• Information geometry and its applications
• Locality of Global Stochastic Interaction in Directed Acyclic Networks
• Riemannian metrics for neural networks. I: Feedforward networks

Cited in

(3)

• Parametrisation independence of the natural gradient in overparametrised systems
• Invariance properties of the natural gradient in overparametrised systems
• On the natural gradient of the evidence lower bound

This page was built for publication: On the locality of the natural gradient for learning in deep Bayesian networks

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6115102)