On neural network kernels and the storage capacity problem

DOI10.1162/NECO_A_01494MaRDI QIDQ5083576zbMATH OpenOpenAlexFDO

Authors Jacob A. Zavatone-Veth, Cengiz Pehlevan

Publication date 20 June 2022

Published in Neural Computation (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2201.04669

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05)

Abstract: In this short note, we reify the connection between work on the storage capacity problem in wide two-layer treelike neural networks and the rapidly-growing body of literature on kernel limits of wide neural networks. Concretely, we observe that the "effective order parameter" studied in the statistical mechanics literature is exactly equivalent to the infinite-width Neural Network Gaussian Process Kernel. This correspondence connects the expressivity and trainability of wide two-layer neural networks.

Recommendations

• scientific article; zbMATH DE number 1928591
• On neuronal capacity
• On the capacity of neural networks with binary weights
• The capacity of feedforward neural networks
• Storage capacity of ultrametric committee machines

Cited in

(4)

• The storage capacity of the Blume–Emery–Griffiths neural network
• Mehler’s Formula, Branching Process, and Compositional Kernels of Deep Neural Networks
• Drifting neuronal representations: bug or feature?
• Memory Capacity of Neural Networks with Threshold and Rectified Linear Unit Activations

This page was built for publication: On neural network kernels and the storage capacity problem

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5083576)