The emergence of a concept in shallow neural networks
From MaRDI portal
(Redirected from Publication:6072447)
Abstract: We consider restricted Boltzmann machine (RBMs) trained over an unstructured dataset made of blurred copies of definite but unavailable ``archetypes and we show that there exists a critical sample size beyond which the RBM can learn archetypes, namely the machine can successfully play as a generative model or as a classifier, according to the operational routine. In general, assessing a critical sample size (possibly in relation to the quality of the dataset) is still an open problem in machine learning. Here, restricting to the random theory, where shallow networks suffice and the grand-mother cell scenario is correct, we leverage the formal equivalence between RBMs and Hopfield networks, to obtain a phase diagram for both the neural architectures which highlights regions, in the space of the control parameters (i.e., number of archetypes, number of neurons, size and quality of the training set), where learning can be accomplished. Our investigations are led by analytical methods based on the statistical-mechanics of disordered systems and results are further corroborated by extensive Monte Carlo simulations.
Recommendations
- Thermodynamics of restricted Boltzmann machines and related learning dynamics
- Pattern reconstruction with restricted Boltzmann machines
- ‘Place-cell’ emergence and learning of invariant data with restricted Boltzmann machines: breaking and dynamical restoration of continuous symmetries in the weight space
- A dynamical mean-field theory for learning in restricted Boltzmann machines
- A topological insight into restricted Boltzmann machines
Cites work
- A mean field view of the landscape of two-layer neural networks
- A transport equation approach for deep neural networks with quenched random weights
- Broken replica symmetry bounds in the mean field spin glass model
- Generalized Guerra's interpolation schemes for dense associative neural networks
- scientific article; zbMATH DE number 2211481 (Why is no real title available?)
- Information, Physics, and Computation
- Machine learning and statistical physics: preface
- Modeling Brain Function
- Neural networks and physical systems with emergent collective computational abilities
- Neural networks retrieving Boolean patterns in a sea of Gaussian ones
- On the effective initialisation for restricted Boltzmann machines via duality with Hopfield model
- On the equivalence of Hopfield networks and Boltzmann machines
- Reducing the Dimensionality of Data with Neural Networks
- Replica symmetry breaking in neural networks: a few steps toward rigorous results
- Statistical mechanics of learning
- Statistical mechanics of unsupervised feature learning in a restricted Boltzmann machine with binary synapses
- The nature of computation
- Thermodynamics of restricted Boltzmann machines and related learning dynamics
- Training Products of Experts by Minimizing Contrastive Divergence
Cited in
(10)- Do ideas have shape? Idea registration as the continuous limit of artificial neural networks
- Hebbian dreaming for small datasets
- Replica symmetry breaking in supervised and unsupervised Hebbian networks
- A spectral approach to Hebbian-like neural networks
- Parallel learning by multitasking neural networks
- Hebbian learning from first principles
- The beneficial role of noises for disentanglement tasks in modular Hebbian networks
- Supervised and unsupervised protocols for hetero-associative neural networks
- Exploring research trends in modern machine learning through a statistical mechanics lens: an Ising-inspired modeling perspective
- Generalized hetero-associative neural networks
This page was built for publication: The emergence of a concept in shallow neural networks
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6072447)
