Improved neural networks based on mutual information via information geometry

DOI10.3390/A12050103MaRDI QIDQ2004876zbMATH OpenOpenAlexFDO

Authors Yanyan Li

Publication date 7 October 2020

Published in Algorithms (Search for Journal in Brave)

Copyright license Creative Commons Attribution 4.0 International

Full work available at URL https://doi.org/10.3390/a12050103

zbMATH Keywords

geodesic lineinformation geometryneural networks

Mathematics Subject Classification ID

Statistical aspects of information-theoretic topics (62B10) Information geometry (statistical aspects) (62B11) Learning and adaptive systems in artificial intelligence (68T05)

Review text

Summary: This paper presents a new algorithm based on the theory of mutual information and information geometry. This algorithm places emphasis on adaptive mutual information estimation and maximum likelihood estimation. With the theory of information geometry, we adjust the mutual information along the geodesic line. Finally, we evaluate our proposal using empirical datasets that are dedicated for classification and regression. The results show that our algorithm contributes to a significant improvement over existing methods.

Recommendations

• Simplified information maximization for improving generalization performance in multilayered neural networks
• An improved Bayesian network structure learning algorithm based on the conditional independence test
• Machine learning with squared-loss mutual information
• An estimate of mutual information that permits closed-form optimisation
• A novel information geometric approach to variable selection in MLP networks

Cites work

• A Fast Learning Algorithm for Deep Belief Nets
• A novel information geometric approach to variable selection in MLP networks
• Asymptotic methods in statistical decision theory
• Conjugate gradient type methods for unsymmetric and inconsistent systems of linear equations
• Evolving Neural Network Ensembles by Minimization of Mutual Information
• scientific article; zbMATH DE number 3522963 (Why is no real title available?)
• scientific article; zbMATH DE number 1055955 (Why is no real title available?)
• Information geometry of estimating functions in semi-parametric statistical models
• Information geometry on hierarchy of probability distributions
• Information geometry. Near randomness and near independence
• Natural gradient learning algorithms for RBF networks
• Numerical recipes. The art of scientific computing.
• Reducing the Dimensionality of Data with Neural Networks
• Science from Fisher Information
• Statistical tests based on geodesic distances
• The relation between information theory and the differential geometry approach to statistics

Cited in

(4)

• Mutual information for explainable deep learning of multiscale systems
• Entropy and mutual information in models of deep neural networks*
• Smoothed noise contrastive mutual information neural estimation
• Bridging algorithmic information theory and machine learning: a new approach to kernel learning

Describes a project that uses

Uses Software

• darch

This page was built for publication: Improved neural networks based on mutual information via information geometry

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2004876)