Dreaming neural networks: rigorous results

DOI10.1088/1742-5468/AB371DzbMATH Open1457.82322arXiv1812.09077OpenAlexW3098745735MaRDI QIDQ5134375FDOQ5134375

Authors: Elena Agliari, Francesco Alemanno, Adriano Barra, Alberto Fachechi 

Publication date: 16 November 2020

Published in: Journal of Statistical Mechanics: Theory and Experiment (Search for Journal in Brave)

Abstract: Recently a daily routine for associative neural networks has been proposed: the network Hebbian-learns during the awake state (thus behaving as a standard Hopfield model), then, during its sleep state, optimizing information storage, it consolidates pure patterns and removes spurious ones: this forces the synaptic matrix to collapse to the projector one (ultimately approaching the Kanter-Sompolinksy model). This procedure keeps the learning Hebbian-based (a biological must) but, by taking advantage of a (properly stylized) sleep phase, still reaches the maximal critical capacity (for symmetric interactions). So far this emerging picture (as well as the bulk of papers on unlearning techniques) was supported solely by mathematically-challenging routes, e.g. mainly replica-trick analysis and numerical simulations: here we rely extensively on Guerra's interpolation techniques developed for neural networks and, in particular, we extend the generalized stochastic stability approach to the case. Confining our description within the replica symmetric approximation (where the previous ones lie), the picture painted regarding this generalization (and the previously existing variations on theme) is here entirely confirmed. Further, still relying on Guerra's schemes, we develop a systematic fluctuation analysis to check where ergodicity is broken (an analysis entirely absent in previous investigations). We find that, as long as the network is awake, ergodicity is bounded by the Amit-Gutfreund-Sompolinsky critical line (as it should), but, as the network sleeps, sleeping destroys spin glass states by extending both the retrieval as well as the ergodic region: after an entire sleeping session the solely surviving regions are retrieval and ergodic ones and this allows the network to achieve the perfect retrieval regime (the number of storable patterns equals the number of neurons in the network).

Full work available at URL: https://arxiv.org/abs/1812.09077

Recommendations

• Hebbian learning, its correlation catastrophe, and unlearning
• Unlearning in the paramagnetic phase of neural network models
• On the unlearning procedure yielding a high-performance associative memory neural network
• Self-organization of day cycle and hierarchical associative memory in live neural network
• Sleeping Our Way to Weight Normalization and Stable Learning

Mathematics Subject Classification ID

Random graphs (graph-theoretic aspects) (05C80) Statistical mechanics of random media, disordered materials (including liquid crystals and spin glasses) (82D30) Neural nets applied to problems in time-dependent statistical mechanics (82C32)

Cites Work

• Universality in bipartite mean field spin glasses
• Equilibrium statistical mechanics of bipartite spin systems
• Title not available (Why is that?)
• Neural networks and physical systems with emergent collective computational abilities
• Rigorous results for the Hopfield model with many patterns
• On the stability of the quenched state in mean-field spin-glass models
• Gibbs states and the set of solutions of random constraint satisfaction problems
• On the equivalence of Hopfield networks and Boltzmann machines
• Spin-glass stochastic stability: a rigorous proof
• Immune networks: multitasking capabilities near saturation
• Title not available (Why is that?)
• On the replica symmetric equations for the Hopfield model
• Gibbs states of the Hopfield model in the regime of perfect memory
• Modeling Brain Function
• Multitasking attractor networks with neuronal threshold noise
• Determining computational complexity from characteristic ``phase transitions
• Parallel retrieval of correlated patterns: from Hopfield networks to Boltzmann machines
• Title not available (Why is that?)
• Statistical mechanics of learning
• The space of interactions in neural network models
• The replica-symmetric solution without replica trick for the Hopfield model
• Gibbs states of the Hopfield model with extensively many patterns.
• The replica symmetric approximation of the analogical neural network
• Neural networks retrieving Boolean patterns in a sea of Gaussian ones
• Exponential inequalities and convergence of moments in the replica-symmetric regime of the Hopfield model.
• About the ergodic regime in the analogical Hopfield neural networks: Moments of the partition function
• Free energies of Boltzmann machines: self-averaging, annealed and replica symmetric approximations in the thermodynamic limit
• A new mechanical approach to handle generalized Hopfield neural networks
• Statistical mechanics of Hopfield-like neural networks with modified interactions
• Replica symmetry breaking in neural networks with modified pseudo-inverse interactions
• Unlearning in the paramagnetic phase of neural network models
• Entropy landscape of solutions in the binary perceptron problem

Cited In (15)

• Generalized Guerra's interpolation schemes for dense associative neural networks
• Replica symmetry breaking in neural networks: a few steps toward rigorous results
• A spectral approach to Hebbian-like neural networks
• Eigenvector dreaming
• Interacting dreaming neural networks
• The relativistic Hopfield model with correlated patterns
• On the effective initialisation for restricted Boltzmann machines via duality with Hopfield model
• PDE/statistical mechanics duality: relation between Guerra's interpolated \(p\)-spin ferromagnets and the Burgers hierarchy
• The decimation scheme for symmetric matrix factorization
• Sleeping Our Way to Weight Normalization and Stable Learning
• Replica symmetry breaking in dense Hebbian neural networks
• Learning and retrieval operational modes for three-layer restricted Boltzmann machines
• Self-organization of day cycle and hierarchical associative memory in live neural network
• Hebbian dreaming for small datasets
• Interpolating between boolean and extremely high noisy patterns through minimal dense associative memories

This page was built for publication: Dreaming neural networks: rigorous results

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5134375)