On neural network kernels and the storage capacity problem

DOI10.1162/NECO_A_01494zbMATH Open1492.68118arXiv2201.04669OpenAlexW4220655231MaRDI QIDQ5083576FDOQ5083576

Authors: Jacob A. Zavatone-Veth, Cengiz Pehlevan 

Publication date: 20 June 2022

Published in: Neural Computation (Search for Journal in Brave)

Abstract: In this short note, we reify the connection between work on the storage capacity problem in wide two-layer treelike neural networks and the rapidly-growing body of literature on kernel limits of wide neural networks. Concretely, we observe that the "effective order parameter" studied in the statistical mechanics literature is exactly equivalent to the infinite-width Neural Network Gaussian Process Kernel. This correspondence connects the expressivity and trainability of wide two-layer neural networks.

Full work available at URL: https://arxiv.org/abs/2201.04669

Recommendations

• scientific article; zbMATH DE number 1928591
• On neuronal capacity
• On the capacity of neural networks with binary weights
• The capacity of feedforward neural networks
• Storage capacity of ultrametric committee machines

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05)

Cited In (4)

• The storage capacity of the Blume–Emery–Griffiths neural network
• Mehler’s Formula, Branching Process, and Compositional Kernels of Deep Neural Networks
• Drifting neuronal representations: bug or feature?
• Memory Capacity of Neural Networks with Threshold and Rectified Linear Unit Activations

This page was built for publication: On neural network kernels and the storage capacity problem

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5083576)