Overparameterized neural networks implement associative memory

DOI10.1073/PNAS.2005013117MaRDI QIDQ5073192zbMATH OpenOpenAlexWikidataFDO

Authors Adityanarayanan Radhakrishnan, Mikhail Belkin, Caroline Uhler

Publication date 5 May 2022

Published in Proceedings of the National Academy of Sciences (Search for Journal in Brave)

Copyright license Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International

Full work available at URL https://arxiv.org/abs/1909.12362

Mathematics Subject Classification ID

Information storage and retrieval of data (68P20) Artificial neural networks and deep learning (68T07) Neural networks for/in biological studies, artificial life and related topics (92B20)

Abstract: Identifying computational mechanisms for memorization and retrieval of data is a long-standing problem at the intersection of machine learning and neuroscience. Our main finding is that standard overparameterized deep neural networks trained using standard optimization methods implement such a mechanism for real-valued data. Empirically, we show that: (1) overparameterized autoencoders store training samples as attractors, and thus, iterating the learned map leads to sample recovery; (2) the same mechanism allows for encoding sequences of examples, and serves as an even more efficient mechanism for memory than autoencoding. Theoretically, we prove that when trained on a single example, autoencoders store the example as an attractor. Lastly, by treating a sequence encoder as a composition of maps, we prove that sequence encoding provides a more efficient mechanism for memory than autoencoding.

Recommendations

• Memory Capacity of Neural Networks with Threshold and Rectified Linear Unit Activations
• scientific article; zbMATH DE number 1989744
• Robust exponential memory in Hopfield networks
• Memorization and Association on a Realistic Neural Model
• On a model of associative memory with huge storage capacity

Cites work

• A Fast Learning Algorithm for Deep Belief Nets
• Deep learning
• Eigenvectors of Orthogonally Decomposable Functions
• Neural networks and physical systems with emergent collective computational abilities
• Nonlinear dynamics and chaos. With applications to physics, biology, chemistry, and engineering
• Reconciling modern machine-learning practice and the classical bias-variance trade-off
• The capacity of the Hopfield associative memory
• The existence of persistent states in the brain
• What regularized auto-encoders learn from the data-generating distribution

Cited in

(4)

• Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation
• Geometry and generalization: eigenvalues as predictors of where a network will fail to generalize
• The energy landscape of the Kuramoto model in random geometric graphs in a circle
• Topological properties of basins of attraction of width bounded autoencoders

This page was built for publication: Overparameterized neural networks implement associative memory

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5073192)