Random neural networks in the infinite width limit as Gaussian processes
Neural networks, originally introduced in the 1940s and 1950s, have become extremely useful as powerful tools for a variety of mathematical problems in many subjects, for example image processing, machine learning, neuroscience, signal processing, manifold learning, language processing, probability and many others. See the paper under review for many references in this regard. This fascinating paper studies an important probabilistic question for a class of networks called fully connected neural networks. They are defined as follows: Fix a positive integer \(L\) as well as \(L+2\) positive integers \(n_0,\dots,n_{L+1}\) and a function \(\sigma: \mathbb R\to \mathbb R\). A fully connected depth-\(L\) neural network with input dimension \(n_0\), output dimension \(n_{L+1}\), hidden layer widths \(n_1...n_{L}\) and nonlinearity \(\sigma\) is any function \(x_{\alpha}\in \mathbb R^{n_0}\to z_{\alpha}^{(L+1)}\in \mathbb R^{n_{L+1}}\) of the form \[ z_{\alpha}^{(l)}=\left\{ \begin{array}{ll} W^{(1)}x_{\alpha}+b^{(1)}, & l=1\\ W^{(l)}\sigma(z_{\alpha}^{(l-1)})+b^{(l)},& l=2,...L+1, \end{array} \right. \] where \(W^{(l)}\in \mathbb R^{n_l\times n_{l-1}}\) are matrices, \(b^{(l)}\in \mathbb R^{n_l}\) are vectors and \(\sigma\) applied to a vector is shorthand for \(\sigma\) applied to each component. The parameters \(L, n_0, . . . , n_{L+1}\) are called the network architecture, and \(z_{\alpha}^{(l)}\in \mathbb R^{n_l}\) called the vector of pre-activations at layer \(l\) corresponding to input \(x_{\alpha}\). A fully connected network with a fixed architecture and given nonlinearity \(\sigma\) is therefore a finite but typically high-dimensional family of functions, parameterized by the network weights (entries of the weight matrices \(W^{(l)}\)) and biases (components of bias vectors \(b^{(l)}\)). This article considers the mapping \(x_{\alpha}\to z_{\alpha}^{(L+1)}\) when the network's weights and biases are chosen independently at random and the hidden layer widths \(n_1, . . . , n_L\) are sent to infinity while the input dimension \(n_0\), output dimension \(n_{L+1}\), and network depth \(L\) are fixed. In this infinite width limit, akin to the large matrix limit in random matrix theory, neural networks with random weights and biases converge to Gaussian processes. The main result of the paper under review is that this holds for general nonlinearities \(\sigma\) and distributions of network weights. The paper is well written with an excellent set of references.
- A note on the Pennington-Worah distribution
- Addition of certain non-commuting random variables
- Bayesian learning for neural networks
- Benign overfitting in linear regression
- Ergodic theory of differentiable dynamical systems
- Estimation of moments of sums of independent real random variables
- Fluctuations of \(\beta\)-Jacobi product processes
- Gaussian fluctuations for products of random matrices
- Lectures on the Combinatorics of Free Probability
- Neural network approximation
- Non-asymptotic results for singular values of Gaussian matrix products
- Noncommuting Random Products
- Nonlinear approximation and (deep) ReLU networks
- On the distribution of the roots of certain symmetric matrices
- Products of many large random matrices and gradients in deep neural networks
- Products of Random Matrices
- Reconciling modern machine-learning practice and the classical bias-variance trade-off
- Surprises in high-dimensional ridgeless least squares interpolation
- The Principles of Deep Learning Theory
- Universal microscopic correlation functions for products of independent Ginibre matrices
- Universal microscopic correlation functions for products of truncated unitary matrices
- Unified field theoretical approach to deep and recurrent neuronal networks
- Wide neural networks with bottlenecks are deep Gaussian processes
- Deep stable neural networks: large-width asymptotics and convergence rates
- -Stable convergence of heavy-/light-tailed infinitely wide neural networks
- Gaussian random field approximation via Stein's method with applications to wide random neural networks
- Infinite-width limit of deep linear neural networks
- Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networks
- LDP for the covariance process in fully connected Gaussian neural networks
- Large deviations of Gaussian neural networks with ReLu activation
- Wide stable neural networks: sample regularity, functional convergence and Bayesian inverse problems
- Spectral complexity of deep neural networks
- Feature learning in finite-width Bayesian deep linear networks with multiple outputs and convolutional layers
- Random ReLU neural networks as non-Gaussian processes
- Quantitative CLTs in deep neural networks
- Trained quantum neural networks are Gaussian processes
- A primer on Bayesian neural networks: review and debates
- Large and moderate deviations for Gaussian neural networks
- Finite-dimensional Gaussian approximation for deep neural networks: universality in random weights
- Fundamental limits of overparametrized shallow neural networks for supervised learning
This page was built for publication: Random neural networks in the infinite width limit as Gaussian processes
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6138923)
