Free energy of multi-layer generalized linear models

DOI10.1007/S00220-022-04630-4MaRDI QIDQ6155616zbMATH OpenOpenAlexFDO

Authors Hongbin Chen, Jiaming Xia

Publication date 5 June 2023

Published in Communications in Mathematical Physics (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2108.12615

Mathematics Subject Classification ID

General first-order partial differential equations and systems of first-order partial differential equations (35Fxx) Equilibrium statistical mechanics (82Bxx) Applications of statistical mechanics to specific types of physical systems (82Dxx)

Abstract: We compute the high-dimensional limit of the free energy associated with a multi-layer generalized linear model. Under certain technical assumptions, we identify the limit in terms of a variational formula. The approach is to first show that the limit is a solution to a Hamilton-Jacobi equation whose initial condition is related to the limiting free energy of a model with one fewer layer. Then, we conclude by an iteration.

Recommendations

• Generalised free energy and active inference
• Inference for Generalized Linear Models via Alternating Directions and Bethe Free Energy Minimization
• Properties of Bethe Free Energies and Message Passing in Gaussian Models
• Penalized maximum likelihood estimation of multi-layered Gaussian graphical models
• Matrix inference and estimation in multi-layer models*
• Strict convexity of the free energy for a class of non-convex gradient models
• Free-energy bounds for hierarchical spin models
• The free energy of a class of Hopfield models
• Measuring the complexity of generalized linear hierarchical models
• An alternative view of variational Bayes and asymptotic approximations of free energy

Cites work

• A mechanical approach to mean field spin models
• Concentration inequalities. A nonasymptotic theory of independence
• Convex Analysis
• Entropy and mutual information in models of deep neural networks*
• Extending the Parisi formula along a Hamilton-Jacobi equation
• Hamilton-Jacobi equations for finite-rank matrix inference
• Hamilton-Jacobi equations for mean-field disordered systems
• Hopf Formula and Multitime Hamilton-Jacobi Equations
• scientific article; zbMATH DE number 1751767 (Why is no real title available?)
• scientific article; zbMATH DE number 5681750 (Why is no real title available?)
• Mutual information for low-rank even-order symmetric tensor estimation
• Nonconvex interactions in mean-field spin glasses
• On Hopf's formulas for solutions of Hamilton-Jacobi equations
• Optimal errors and phase transitions in high-dimensional generalized linear models
• Replica symmetry breaking in mean-field spin glasses through the Hamilton-Jacobi technique
• The adaptive interpolation method for proving replica formulas. Applications to the Curie–Weiss and Wigner spike models
• The adaptive interpolation method: a simple scheme to prove replica formulas in Bayesian inference

Cited in

(8)

• Mutual information for the sparse stochastic block model
• Infinite-dimensional Hamilton-Jacobi equations for statistical inference on sparse graphs
• Envelope representation of Hamilton–Jacobi equations from spin glasses
• Color symmetry and ferromagnetism in Potts spin glass
• Hamilton-Jacobi equations from mean-field spin glasses
• On free energy of non-convex multi-species spin glasses
• Differentiability and overlap concentration in optimal Bayesian inference
• Fundamental limits of overparametrized shallow neural networks for supervised learning

This page was built for publication: Free energy of multi-layer generalized linear models

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6155616)