Random neural networks in the infinite width limit as Gaussian processes

DOI10.1214/23-AAP1933arXiv2107.01562MaRDI QIDQ6138923FDOQ6138923

Authors: Boris Hanin 

Publication date: 16 January 2024

Published in: The Annals of Applied Probability (Search for Journal in Brave)

Abstract: This article gives a new proof that fully connected neural networks with random weights and biases converge to Gaussian processes in the regime where the input dimension, output dimension, and depth are kept fixed, while the hidden layer widths tend to infinity. Unlike prior work, convergence is shown assuming only moment conditions for the distribution of weights and for quite general non-linearities.

Full work available at URL: https://arxiv.org/abs/2107.01562

zbMATH Keywords

Gaussian processesneural networkslimit theorems

Mathematics Subject Classification ID

Gaussian processes (60G15) Artificial neural networks and deep learning (68T07)

Cites Work

• Lectures on the Combinatorics of Free Probability
• Ergodic theory of differentiable dynamical systems
• Universal microscopic correlation functions for products of truncated unitary matrices
• Addition of certain non-commuting random variables
• Universal microscopic correlation functions for products of independent Ginibre matrices
• Products of Random Matrices
• Noncommuting Random Products
• On the distribution of the roots of certain symmetric matrices
• Bayesian learning for neural networks
• Estimation of moments of sums of independent real random variables
• Fluctuations of \(\beta\)-Jacobi product processes
• Reconciling modern machine-learning practice and the classical bias–variance trade-off
• Neural network approximation
• Non-asymptotic results for singular values of Gaussian matrix products
• Benign overfitting in linear regression
• Nonlinear approximation and (deep) ReLU networks
• Products of many large random matrices and gradients in deep neural networks
• A note on the Pennington-Worah distribution
• Gaussian fluctuations for products of random matrices
• Surprises in high-dimensional ridgeless least squares interpolation
• The Principles of Deep Learning Theory

Cited In (1)

• Gaussian random field approximation via Stein's method with applications to wide random neural networks

This page was built for publication: Random neural networks in the infinite width limit as Gaussian processes

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6138923)