Annealing and replica-symmetry in deep Boltzmann machines
From MaRDI portal
Publication:2194174
DOI10.1007/S10955-020-02495-2zbMATH Open1453.82087arXiv2001.07714OpenAlexW3002645806MaRDI QIDQ2194174FDOQ2194174
Authors: Diego Alberici, Adriano Barra, P. Contucci, Emanuele Mingione 
Publication date: 25 August 2020
Published in: Journal of Statistical Physics (Search for Journal in Brave)
Abstract: In this paper we study the properties of the quenched pressure of a multi-layer spin-glass model (a deep Boltzmann Machine in artificial intelligence jargon) whose pairwise interactions are allowed between spins lying in adjacent layers and not inside the same layer nor among layers at distance larger than one. We prove a theorem that bounds the quenched pressure of such a K-layer machine in terms of K Sherrington-Kirkpatrick spin glasses and use it to investigate its annealed region. The replica-symmetric approximation of the quenched pressure is identified and its relation to the annealed one is considered. The paper also presents some observation on the model's architectural structure related to machine learning. Since escaping the annealed region is mandatory for a meaningful training, by squeezing such region we obtain thermodynamical constraints on the form factors. Remarkably, its optimal escape is achieved by requiring the last layer to scale sub-linearly in the network size.
Full work available at URL: https://arxiv.org/abs/2001.07714
Recommendations
- Deep Boltzmann machines: rigorous results at arbitrary depth
- A transport equation approach for deep neural networks with quenched random weights
- The solution of the deep Boltzmann machine on the Nishimori line
- Minimax formula for the replica symmetric free energy of deep restricted Boltzmann machines
- Boltzmann Machines with Bounded Continuous Random Variables
Learning and adaptive systems in artificial intelligence (68T05) Statistical mechanics of random media, disordered materials (including liquid crystals and spin glasses) (82D30)
Cites Work
- Equilibrium statistical mechanics of bipartite spin systems
- Title not available (Why is that?)
- Title not available (Why is that?)
- The thermodynamic limit in mean field spin glass models
- Broken replica symmetry bounds in the mean field spin glass model
- The Sherrington-Kirkpatrick model
- Some rigorous results on the Sherrington-Kirkpatrick spin glass model.
- Multi-species mean field spin glasses. Rigorous results
- The free energy in a multi-species Sherrington-Kirkpatrick model
- On the equivalence of Hopfield networks and Boltzmann machines
- Information, Physics, and Computation
- Perspectives on spin glasses
- Title not available (Why is that?)
- Statistical Physics of Spin Glasses and Information Processing
- Modeling Brain Function
- Free energy and complexity of spherical bipartite models
- Non-convex multi-species Hopfield models
- Optimal errors and phase transitions in high-dimensional generalized linear models
Cited In (16)
- Fluctuation results for multi-species Sherrington-Kirkpatrick model in the replica symmetric regime
- Replica symmetry breaking in supervised and unsupervised Hebbian networks
- Fluctuations for the bipartite Sherrington-Kirkpatrick model
- A transport equation approach for deep neural networks with quenched random weights
- Thermodynamics of bidirectional associative memories
- Deep Boltzmann machines: rigorous results at arbitrary depth
- Dense Hebbian neural networks: a replica symmetric picture of supervised learning
- Minimax formula for the replica symmetric free energy of deep restricted Boltzmann machines
- Free energy in multi-species mixed \(p\)-spin spherical models
- The solution of the deep Boltzmann machine on the Nishimori line
- A study on the characteristics in a symmetry Boltzmann machine composed of two Boltzmann machines
- The multi-species mean-field spin-glass on the Nishimori line
- Deep learning the Ising model near criticality
- Hopfield model with planted patterns: a teacher-student self-supervised learning model
- On the free energy of vector spin glasses with nonconvex interactions
- Free energies of Boltzmann machines: self-averaging, annealed and replica symmetric approximations in the thermodynamic limit
This page was built for publication: Annealing and replica-symmetry in deep Boltzmann machines
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2194174)
