High-temperature expansions and message passing algorithms
From MaRDI portal
Publication:5854076
Abstract: Improved mean-field technics are a central theme of statistical physics methods applied to inference and learning. We revisit here some of these methods using high-temperature expansions for disordered systems initiated by Plefka, Georges and Yedidia. We derive the Gibbs free entropy and the subsequent self-consistent equations for a generic class of statistical models with correlated matrices and show in particular that many classical approximation schemes, such as adaptive TAP, Expectation-Consistency, or the approximations behind the Vector Approximate Message Passing algorithm all rely on the same assumptions, that are also at the heart of high-temperature expansions. We focus on the case of rotationally invariant random coupling matrices in the `high-dimensional' limit in which the number of samples and the dimension are both large, but with a fixed ratio. This encapsulates many widely studied models, such as Restricted Boltzmann Machines or Generalized Linear Models with correlated data matrices. In this general setting, we show that all the approximation schemes described before are equivalent, and we conjecture that they are exact in the thermodynamic limit in the replica symmetric phases. We achieve this conclusion by resummation of the infinite perturbation series, which generalizes a seminal result of Parisi and Potters. A rigorous derivation of this conjecture is an interesting mathematical challenge. On the way to these conclusions, we uncover several diagrammatical results in connection with free probability and random matrix theory, that are interesting independently of the rest of our work.
Recommendations
- Mean-field inference methods for neural networks
- A Unifying Tutorial on Approximate Message Passing
- Approximate inference in Boltzmann machines
- Mean field asymptotics in high-dimensional statistics: from exact results to efficient algorithms
- A dynamical mean-field theory for learning in restricted Boltzmann machines
Cites work
- scientific article; zbMATH DE number 3945130 (Why is no real title available?)
- scientific article; zbMATH DE number 1273988 (Why is no real title available?)
- scientific article; zbMATH DE number 2174437 (Why is no real title available?)
- A Fourier view on the \(R\)-transform and related asymptotics of spherical integrals
- A theory of solving TAP equations for Ising models with general invariant random matrices
- Advanced mean field methods. Theory and practice
- An introduction to random matrices
- Compressed sensing
- DISTRIBUTION OF EIGENVALUES FOR SOME SETS OF RANDOM MATRICES
- Differential Operators on a Semisimple Lie Algebra
- Expectation consistent approximate inference
- Graphical models, exponential families, and variational inference
- Information, Physics, and Computation
- Integration with respect to the Haar measure on unitary, orthogonal and symplectic group
- Mean-field equations for spin models with orthogonal interaction matrices
- Mean-field theory for a spin-glass model of neural networks: TAP free energy and the paramagnetic to spin-glass transition
- Neural networks and physical systems with emergent collective computational abilities
- New scaling of Itzykson-Zuber integrals
- Optimal errors and phase transitions in high-dimensional generalized linear models
- Perceptron capacity revisited: classification ability for correlated patterns
- Random Matrix Theory and Wireless Communications
- Replica field theory for deterministic models. II. A non-random spin glass with glassy behaviour
- Replica field theory for deterministic models: I. Binary sequences with low autocorrelation
- The jamming transition in high dimension: an analytical study of the TAP equations and the effective thermodynamic potential
- The planar approximation. II
- Vector Approximate Message Passing
Cited in
(9)- Perturbative construction of mean-field equations in extensive-rank matrix factorization and denoising
- Approximate message passing algorithms for rotationally invariant matrices
- Structured random matrices and cyclic cumulants: a free probability approach
- Mean-field inference methods for neural networks
- The replica-symmetric free energy for Ising spin glasses with orthogonally invariant couplings
- Solving the spherical \textbf{\(p\)}-spin model with the cavity method: equivalence with the replica results
- Matrix denoising: Bayes-optimal estimators via low-degree polynomials
- A dynamical mean-field theory for learning in restricted Boltzmann machines
- Marginals of a spherical spin Glass model with correlated disorder
This page was built for publication: High-temperature expansions and message passing algorithms
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5854076)
