A dynamical mean-field theory for learning in restricted Boltzmann machines
From MaRDI portal
Publication:5857421
Abstract: We define a message-passing algorithm for computing magnetizations in Restricted Boltzmann machines, which are Ising models on bipartite graphs introduced as neural network models for probability distributions over spin configurations. To model nontrivial statistical dependencies between the spins' couplings, we assume that the rectangular coupling matrix is drawn from an arbitrary bi-rotation invariant random matrix ensemble. Using the dynamical functional method of statistical mechanics we exactly analyze the dynamics of the algorithm in the large system limit. We prove the global convergence of the algorithm under a stability criterion and compute asymptotic convergence rates showing excellent agreement with numerical simulations.
Recommendations
- Thermodynamics of restricted Boltzmann machines and related learning dynamics
- Restricted Boltzmann machines: introduction and review
- Training restricted Boltzmann machines: an introduction
- Statistical mechanics of unsupervised feature learning in a restricted Boltzmann machine with binary synapses
- Learning restricted Boltzmann machines via influence maximization
Cites work
- scientific article; zbMATH DE number 1273988 (Why is no real title available?)
- A theory of solving TAP equations for Ising models with general invariant random matrices
- An iterative construction of solutions of the TAP equations for the Sherrington-Kirkpatrick model
- Analysis of Bayesian inference algorithms by the dynamical functional approach
- Expectation consistent approximate inference
- High-temperature expansions and message passing algorithms
- Information, Physics, and Computation
- Integration of invariant matrices and moments of inverses of Ginibre and Wishart matrices
- Introduction to random matrices. Theory and practice
- Mean-field inference methods for neural networks
- New method for studying the dynamics of disordered spin systems without finite-size effects
- New scaling of Itzykson-Zuber integrals
- Random matrix methods for wireless communications.
- Rectangular \(R\)-transform as the limit of rectangular spherical integrals
- Rigorous Dynamics of Expectation-Propagation-Based Signal Recovery from Unitarily Invariant Measurements
- Statistical Physics of Spin Glasses and Information Processing
- The Dynamics of Message Passing on Dense Graphs, with Applications to Compressed Sensing
- The planar approximation. II
- The space of interactions in neural network models
- Thermodynamics of restricted Boltzmann machines and related learning dynamics
- Training Products of Experts by Minimizing Contrastive Divergence
- Vector Approximate Message Passing
Cited in
(20)- Boltzmann Machine and Mean-Field Approximation for Structured Sparse Decompositions
- Thermodynamics of restricted Boltzmann machines and related learning dynamics
- Learning and Inference in Sparse Coding Models With Langevin Dynamics
- Approximate message passing algorithms for rotationally invariant matrices
- Analysis of Bayesian inference algorithms by the dynamical functional approach
- Minimax formula for the replica symmetric free energy of deep restricted Boltzmann machines
- The solution of the deep Boltzmann machine on the Nishimori line
- Gaussian-spherical restricted Boltzmann machines
- Universality of approximate message passing algorithms and tensor networks
- Dynamical analysis of contrastive divergence learning: restricted Boltzmann machines with Gaussian visible units
- The flip-the-state transition operator for restricted Boltzmann machines
- Learning restricted Boltzmann machines via influence maximization
- Stochastic complexity and generalization error of a restricted Boltzmann machine in Bayesian estimation
- High-temperature expansions and message passing algorithms
- Statistical mechanics of unsupervised feature learning in a restricted Boltzmann machine with binary synapses
- A bound for the convergence rate of parallel tempering for sampling restricted Boltzmann machines
- Analysis of random sequential message passing algorithms for approximate inference
- ‘Place-cell’ emergence and learning of invariant data with restricted Boltzmann machines: breaking and dynamical restoration of continuous symmetries in the weight space
- The emergence of a concept in shallow neural networks
- Learning large \(Q\)-matrix by restricted Boltzmann machines
This page was built for publication: A dynamical mean-field theory for learning in restricted Boltzmann machines
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5857421)
