A random matrix approach to neural networks

DOI10.1214/17-AAP1328MaRDI QIDQ1650102zbMATH OpenOpenAlexWikidataFDO

Authors Cosme Louart, Zhenyu Liao, R. Couillet

Publication date 29 June 2018

Published in The Annals of Applied Probability (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1702.05419, https://projecteuclid.org/euclid.aoap/1523433634

zbMATH Keywords

neural networksrandom feature mapsrandom matrix theory

Mathematics Subject Classification ID

Random matrices (algebraic aspects) (15B52) Random matrices (probabilistic aspects) (60B20) Neural nets and related approaches to inference from stochastic processes (62M45) Neural networks for/in biological studies, artificial life and related topics (92B20)

Abstract: This article studies the Gram random matrix model $G=frac1TSigm{a}^{mT}Sigma$, $Sigma=sigma(WX)$, classically found in the analysis of random feature maps and random neural networks, where $X=[x_1,ldots,x_T]in{mathbbR}^{pimesT}$ is a (data) matrix of bounded norm, $Win{mathbbR}^{nimesp}$ is a matrix of independent zero-mean unit variance entries, and $sigma:mathbbRomathbbR$ is a Lipschitz continuous (activation) function --- $sigma(WX)$ being understood entry-wise. By means of a key concentration of measure lemma arising from non-asymptotic random matrix arguments, we prove that, as $n,p,T$ grow large at the same rate, the resolvent $Q=(G+gamma{I}_T{)}^{-1}$, for $gamma>0$, has a similar behavior as that met in sample covariance matrix models, involving notably the moment $Phi=fracTnmathbbE[G]$, which provides in passing a deterministic equivalent for the empirical spectral measure of $G$. Application-wise, this result enables the estimation of the asymptotic performance of single-layer random neural networks. This in turn provides practical insights into the underlying mechanisms into play in random neural networks, entailing several unexpected consequences, as well as a fast practical means to tune the network hyperparameters.

Recommendations

• Nonlinear random matrix theory for deep learning
• Eigenvalue distribution of some nonlinear models of random matrices
• On random matrices arising in deep neural networks: General I.I.D. case
• On Random Matrices Arising in Deep Neural Networks. Gaussian Case
• Products of many large random matrices and gradients in deep neural networks

Cites work

• A Central Limit Theorem for the SINR at the LMMSE Estimator Output for Large-Dimensional Signals
• A Large Dimensional Analysis of Least Squares Support Vector Machines
• Almost sure localization of the eigenvalues in a Gaussian information plus noise model. Application to the spiked models.
• AN EXAMPLE IN THE THEORY OF THE SPECTRUM OF A FUNCTION
• Analysis of the limiting spectral distribution of large dimensional random matrices
• Concentration of measure and spectra of random matrices: applications to correlation matrices, elliptical distributions and beyond
• Deep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy?
• DISTRIBUTION OF EIGENVALUES FOR SOME SETS OF RANDOM MATRICES
• Eigenvalue distribution of large random matrices
• Hanson-Wright inequality and sub-Gaussian concentration
• scientific article; zbMATH DE number 876688 (Why is no real title available?)
• Kernel spectral clustering of large dimensional data
• Multilayer feedforward networks are universal approximators
• No eigenvalues outside the support of the limiting spectral distribution of large-dimensional sample covariance matrices
• On the empirical distribution of eigenvalues of a class of large dimensional random matrices
• On the signal-to-interference ratio of CDMA systems in wireless communications
• Random Beamforming Over Quasi-Static and Fading Channels: A Deterministic Equivalent Approach
• Spectral analysis of large dimensional random matrices
• The random matrix regime of Maronna's M-estimator with elliptically distributed samples
• The singular values and vectors of low rank perturbations of large rectangular random matrices
• The spectrum of kernel random matrices

Cited in

(37)

• Eigenvalue distribution of some nonlinear models of random matrices
• Products of many large random matrices and gradients in deep neural networks
• A note on the Pennington-Worah distribution
• Free dynamics of feature learning processes
• Halting time is predictable for large models: a universality property and average-case analysis
• scientific article; zbMATH DE number 7038932 (Why is no real title available?)
• scientific article; zbMATH DE number 999632 (Why is no real title available?)
• Random interactions in higher order neural networks
• On Random Matrices Arising in Deep Neural Networks. Gaussian Case
• A random matrix analysis of random Fourier features: beyond the Gaussian kernel, a precise phase transition, and the corresponding double descent*
• Generalisation error in learning with random features and the hidden manifold model*
• On the empirical spectral distribution for certain models related to sample covariance matrices with different correlations
• What causes the test error? Going beyond bias-variance via ANOVA
• FAST NON-NEGATIVE LEAST-SQUARES LEARNING IN THE RANDOM NEURAL NETWORK
• scientific article; zbMATH DE number 2233617 (Why is no real title available?)
• Nonlinear random matrix theory for deep learning
• Deep learning: a statistical viewpoint
• Deep learning in random neural fields: numerical experiments via neural tangent kernel
• Large-dimensional random matrix theory and its applications in deep learning and wireless communications
• Designing universal causal deep learning models: The geometric (Hyper)transformer
• The curse of overparametrization in adversarial training: precise analysis of robust generalization for random features regression
• Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networks
• Dyson equation for correlated linearizations and test error of random features regression
• Spectra of large dimensional random Gram matrices under partial dependence
• Global law of conjugate kernel random matrices with heavy-tailed weights
• Homogenization of SGD in high-dimensions: exact dynamics and generalization properties
• Extremal eigenvalues of random kernel matrices with polynomial scaling
• An equivalence principle for the spectrum of random inner-product kernel matrices with polynomial scalings
• High-dimensional learning of narrow neural networks
• Hitting the high-dimensional notes: an ODE for SGD learning dynamics on GLMs and multi-index models
• Injectivity of ReLU networks: perspectives from statistical physics
• Universality laws for Gaussian mixtures in generalized linear models
• Bayes-optimal learning of deep random networks of extensive-width
• Feature learning and generalization error analysis of two-layer linear neural networks for high-dimensional inputs
• Universality of kernel random matrices and kernel regression in the quadratic regime
• Scaling and renormalization in high-dimensional regression
• Pseudo-maximum likelihood theory for high-dimensional rank one inference

Describes a project that uses

Uses Software

• MNIST
• ImageNet
• AlexNet

This page was built for publication: A random matrix approach to neural networks

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q1650102)