Overparameterized neural networks implement associative memory

DOI10.1073/PNAS.2005013117zbMATH Open1485.68236arXiv1909.12362OpenAlexW3093151833WikidataQ100693711 ScholiaQ100693711MaRDI QIDQ5073192FDOQ5073192

Authors: Adityanarayanan Radhakrishnan, Mikhail Belkin, Caroline Uhler 

Publication date: 5 May 2022

Published in: Proceedings of the National Academy of Sciences (Search for Journal in Brave)

Abstract: Identifying computational mechanisms for memorization and retrieval of data is a long-standing problem at the intersection of machine learning and neuroscience. Our main finding is that standard overparameterized deep neural networks trained using standard optimization methods implement such a mechanism for real-valued data. Empirically, we show that: (1) overparameterized autoencoders store training samples as attractors, and thus, iterating the learned map leads to sample recovery; (2) the same mechanism allows for encoding sequences of examples, and serves as an even more efficient mechanism for memory than autoencoding. Theoretically, we prove that when trained on a single example, autoencoders store the example as an attractor. Lastly, by treating a sequence encoder as a composition of maps, we prove that sequence encoding provides a more efficient mechanism for memory than autoencoding.

Full work available at URL: https://arxiv.org/abs/1909.12362

Recommendations

• Memory Capacity of Neural Networks with Threshold and Rectified Linear Unit Activations
• scientific article; zbMATH DE number 1989744
• Robust exponential memory in Hopfield networks
• Memorization and Association on a Realistic Neural Model
• On a model of associative memory with huge storage capacity

Mathematics Subject Classification ID

Information storage and retrieval of data (68P20) Artificial neural networks and deep learning (68T07) Neural networks for/in biological studies, artificial life and related topics (92B20)

Cites Work

• Deep learning
• Neural networks and physical systems with emergent collective computational abilities
• Eigenvectors of Orthogonally Decomposable Functions
• The capacity of the Hopfield associative memory
• Nonlinear dynamics and chaos. With applications to physics, biology, chemistry, and engineering
• A Fast Learning Algorithm for Deep Belief Nets
• The existence of persistent states in the brain
• Reconciling modern machine-learning practice and the classical bias-variance trade-off
• What regularized auto-encoders learn from the data-generating distribution

Cited In (4)

• Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation
• Topological properties of basins of attraction of width bounded autoencoders
• Geometry and generalization: eigenvalues as predictors of where a network will fail to generalize
• The energy landscape of the Kuramoto model in random geometric graphs in a circle

This page was built for publication: Overparameterized neural networks implement associative memory

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5073192)