Annealing and replica-symmetry in deep Boltzmann machines
From MaRDI portal
Publication:2194174
Abstract: In this paper we study the properties of the quenched pressure of a multi-layer spin-glass model (a deep Boltzmann Machine in artificial intelligence jargon) whose pairwise interactions are allowed between spins lying in adjacent layers and not inside the same layer nor among layers at distance larger than one. We prove a theorem that bounds the quenched pressure of such a K-layer machine in terms of K Sherrington-Kirkpatrick spin glasses and use it to investigate its annealed region. The replica-symmetric approximation of the quenched pressure is identified and its relation to the annealed one is considered. The paper also presents some observation on the model's architectural structure related to machine learning. Since escaping the annealed region is mandatory for a meaningful training, by squeezing such region we obtain thermodynamical constraints on the form factors. Remarkably, its optimal escape is achieved by requiring the last layer to scale sub-linearly in the network size.
Recommendations
- Deep Boltzmann machines: rigorous results at arbitrary depth
- A transport equation approach for deep neural networks with quenched random weights
- The solution of the deep Boltzmann machine on the Nishimori line
- Minimax formula for the replica symmetric free energy of deep restricted Boltzmann machines
- Boltzmann Machines with Bounded Continuous Random Variables
Cites work
- scientific article; zbMATH DE number 1273988 (Why is no real title available?)
- scientific article; zbMATH DE number 1952026 (Why is no real title available?)
- scientific article; zbMATH DE number 2211481 (Why is no real title available?)
- Broken replica symmetry bounds in the mean field spin glass model
- Equilibrium statistical mechanics of bipartite spin systems
- Free energy and complexity of spherical bipartite models
- Information, Physics, and Computation
- Modeling Brain Function
- Multi-species mean field spin glasses. Rigorous results
- Non-convex multi-species Hopfield models
- On the equivalence of Hopfield networks and Boltzmann machines
- Optimal errors and phase transitions in high-dimensional generalized linear models
- Perspectives on spin glasses
- Some rigorous results on the Sherrington-Kirkpatrick spin glass model.
- Statistical Physics of Spin Glasses and Information Processing
- The Sherrington-Kirkpatrick model
- The free energy in a multi-species Sherrington-Kirkpatrick model
- The thermodynamic limit in mean field spin glass models
Cited in
(16)- Free energies of Boltzmann machines: self-averaging, annealed and replica symmetric approximations in the thermodynamic limit
- Fluctuation results for multi-species Sherrington-Kirkpatrick model in the replica symmetric regime
- Replica symmetry breaking in supervised and unsupervised Hebbian networks
- Fluctuations for the bipartite Sherrington-Kirkpatrick model
- A transport equation approach for deep neural networks with quenched random weights
- Deep Boltzmann machines: rigorous results at arbitrary depth
- Thermodynamics of bidirectional associative memories
- Dense Hebbian neural networks: a replica symmetric picture of supervised learning
- Minimax formula for the replica symmetric free energy of deep restricted Boltzmann machines
- Free energy in multi-species mixed \(p\)-spin spherical models
- The solution of the deep Boltzmann machine on the Nishimori line
- A study on the characteristics in a symmetry Boltzmann machine composed of two Boltzmann machines
- The multi-species mean-field spin-glass on the Nishimori line
- Deep learning the Ising model near criticality
- Hopfield model with planted patterns: a teacher-student self-supervised learning model
- On the free energy of vector spin glasses with nonconvex interactions
This page was built for publication: Annealing and replica-symmetry in deep Boltzmann machines
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2194174)
