Random neural networks in the infinite width limit as Gaussian processes

DOI10.1214/23-AAP1933MaRDI QIDQ6138923FDOQ6138923

Authors: Boris Hanin 

Publication date: 16 January 2024

Published in: The Annals of Applied Probability (Search for Journal in Brave)

Abstract: This article gives a new proof that fully connected neural networks with random weights and biases converge to Gaussian processes in the regime where the input dimension, output dimension, and depth are kept fixed, while the hidden layer widths tend to infinity. Unlike prior work, convergence is shown assuming only moment conditions for the distribution of weights and for quite general non-linearities.

Full work available at URL: https://arxiv.org/abs/2107.01562

Recommendations

• Infinite-width limit of deep linear neural networks
• Mean field analysis of neural networks: a central limit theorem
• Wide neural networks with bottlenecks are deep Gaussian processes

zbMATH Keywords

Gaussian processesneural networkslimit theorems

Mathematics Subject Classification ID

Gaussian processes (60G15) Artificial neural networks and deep learning (68T07)

Cites Work

• A note on the Pennington-Worah distribution
• Addition of certain non-commuting random variables
• Bayesian learning for neural networks
• Benign overfitting in linear regression
• Ergodic theory of differentiable dynamical systems
• Estimation of moments of sums of independent real random variables
• Fluctuations of \(\beta\)-Jacobi product processes
• Gaussian fluctuations for products of random matrices
• Lectures on the Combinatorics of Free Probability
• Neural network approximation
• Non-asymptotic results for singular values of Gaussian matrix products
• Noncommuting Random Products
• Nonlinear approximation and (deep) ReLU networks
• On the distribution of the roots of certain symmetric matrices
• Products of Random Matrices
• Products of many large random matrices and gradients in deep neural networks
• Reconciling modern machine-learning practice and the classical bias-variance trade-off
• Surprises in high-dimensional ridgeless least squares interpolation
• The Principles of Deep Learning Theory
• Universal microscopic correlation functions for products of independent Ginibre matrices
• Universal microscopic correlation functions for products of truncated unitary matrices

Cited In (6)

• Deep stable neural networks: large-width asymptotics and convergence rates
• Gaussian random field approximation via Stein's method with applications to wide random neural networks
• Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networks
• Infinite-width limit of deep linear neural networks
• -Stable convergence of heavy-/light-tailed infinitely wide neural networks
• Unified field theoretical approach to deep and recurrent neuronal networks

This page was built for publication: Random neural networks in the infinite width limit as Gaussian processes

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6138923)