A random matrix approach to neural networks

DOI10.1214/17-AAP1328zbMATH Open1391.60010arXiv1702.05419OpenAlexW2963650649WikidataQ130038206 ScholiaQ130038206MaRDI QIDQ1650102FDOQ1650102

Authors: Cosme Louart, Zhenyu Liao, R. Couillet 

Publication date: 29 June 2018

Published in: The Annals of Applied Probability (Search for Journal in Brave)

Abstract: This article studies the Gram random matrix model $G=frac1TSigm{a}^{mT}Sigma$, $Sigma=sigma(WX)$, classically found in the analysis of random feature maps and random neural networks, where $X=[x_1,ldots,x_T]in{mathbbR}^{pimesT}$ is a (data) matrix of bounded norm, $Win{mathbbR}^{nimesp}$ is a matrix of independent zero-mean unit variance entries, and $sigma:mathbbRomathbbR$ is a Lipschitz continuous (activation) function --- $sigma(WX)$ being understood entry-wise. By means of a key concentration of measure lemma arising from non-asymptotic random matrix arguments, we prove that, as $n,p,T$ grow large at the same rate, the resolvent $Q=(G+gamma{I}_T{)}^{-1}$, for $gamma>0$, has a similar behavior as that met in sample covariance matrix models, involving notably the moment $Phi=fracTnmathbbE[G]$, which provides in passing a deterministic equivalent for the empirical spectral measure of $G$. Application-wise, this result enables the estimation of the asymptotic performance of single-layer random neural networks. This in turn provides practical insights into the underlying mechanisms into play in random neural networks, entailing several unexpected consequences, as well as a fast practical means to tune the network hyperparameters.

Full work available at URL: https://arxiv.org/abs/1702.05419

Recommendations

• Nonlinear random matrix theory for deep learning
• Eigenvalue distribution of some nonlinear models of random matrices
• On random matrices arising in deep neural networks: General I.I.D. case
• On Random Matrices Arising in Deep Neural Networks. Gaussian Case
• Products of many large random matrices and gradients in deep neural networks

zbMATH Keywords

random matrix theoryneural networksrandom feature maps

Mathematics Subject Classification ID

Random matrices (algebraic aspects) (15B52) Random matrices (probabilistic aspects) (60B20) Neural nets and related approaches to inference from stochastic processes (62M45) Neural networks for/in biological studies, artificial life and related topics (92B20)

Cites Work

• Spectral analysis of large dimensional random matrices
• Title not available (Why is that?)
• DISTRIBUTION OF EIGENVALUES FOR SOME SETS OF RANDOM MATRICES
• Hanson-Wright inequality and sub-Gaussian concentration
• The singular values and vectors of low rank perturbations of large rectangular random matrices
• No eigenvalues outside the support of the limiting spectral distribution of large-dimensional sample covariance matrices
• Analysis of the limiting spectral distribution of large dimensional random matrices
• Eigenvalue distribution of large random matrices
• Multilayer feedforward networks are universal approximators
• AN EXAMPLE IN THE THEORY OF THE SPECTRUM OF A FUNCTION
• Concentration of measure and spectra of random matrices: applications to correlation matrices, elliptical distributions and beyond
• On the empirical distribution of eigenvalues of a class of large dimensional random matrices
• The random matrix regime of Maronna's M-estimator with elliptically distributed samples
• Kernel spectral clustering of large dimensional data
• A Central Limit Theorem for the SINR at the LMMSE Estimator Output for Large-Dimensional Signals
• The spectrum of kernel random matrices
• Almost sure localization of the eigenvalues in a Gaussian information plus noise model. Application to the spiked models.
• On the signal-to-interference ratio of CDMA systems in wireless communications
• A Large Dimensional Analysis of Least Squares Support Vector Machines
• Random Beamforming Over Quasi-Static and Fading Channels: A Deterministic Equivalent Approach
• Deep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy?

Cited In (21)

• A random matrix analysis of random Fourier features: beyond the Gaussian kernel, a precise phase transition, and the corresponding double descent*
• On the empirical spectral distribution for certain models related to sample covariance matrices with different correlations
• Random interactions in higher order neural networks
• Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networks
• On Random Matrices Arising in Deep Neural Networks. Gaussian Case
• Title not available (Why is that?)
• Large-dimensional random matrix theory and its applications in deep learning and wireless communications
• Deep learning in random neural fields: numerical experiments via neural tangent kernel
• Nonlinear random matrix theory for deep learning
• Products of many large random matrices and gradients in deep neural networks
• Designing universal causal deep learning models: The geometric (Hyper)transformer
• Free dynamics of feature learning processes
• Title not available (Why is that?)
• Deep learning: a statistical viewpoint
• Generalisation error in learning with random features and the hidden manifold model*
• FAST NON-NEGATIVE LEAST-SQUARES LEARNING IN THE RANDOM NEURAL NETWORK
• Eigenvalue distribution of some nonlinear models of random matrices
• Title not available (Why is that?)
• Halting time is predictable for large models: a universality property and average-case analysis
• A note on the Pennington-Worah distribution
• The curse of overparametrization in adversarial training: precise analysis of robust generalization for random features regression

Uses Software

• MNIST
• ImageNet
• AlexNet

This page was built for publication: A random matrix approach to neural networks

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q1650102)