Eigenvalue distribution of some nonlinear models of random matrices

DOI10.1214/21-EJP699MaRDI QIDQ2076620zbMATH OpenOpenAlexFDO

Authors S. Péché, Lucas Benigni

Publication date 22 February 2022

Published in Electronic Journal of Probability (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1904.03090

zbMATH Keywords

machine learningneural networksrandom matrices

Mathematics Subject Classification ID

Random matrices (algebraic aspects) (15B52) Random matrices (probabilistic aspects) (60B20) Neural nets and related approaches to inference from stochastic processes (62M45)

Abstract: This paper is concerned with the asymptotic empirical eigenvalue distribution of a non linear random matrix ensemble. More precisely we consider $M=frac1mY{Y}^{\ast }$ with $Y=f(WX)$ where $W$ and $X$ are random rectangular matrices with i.i.d. centered entries. The function $f$ is applied pointwise and can be seen as an activation function in (random) neural networks. We compute the asymptotic empirical distribution of this ensemble in the case where $W$ and $X$ have sub-Gaussian tails and $f$ is real analytic. This extends a previous result where the case of Gaussian matrices $W$ and $X$ is considered. We also investigate the same questions in the multi-layer case, regarding neural network applications.

Recommendations

• Nonlinear random matrix theory for deep learning
• A note on the Pennington-Worah distribution
• On random matrices arising in deep neural networks: General I.I.D. case
• A random matrix approach to neural networks
• Eigenvalue distribution of large random matrices arising in deep neural networks: Orthogonal case

Cites work

• A note on the Pennington-Worah distribution
• A random matrix approach to neural networks
• COMBINATORIAL PROBLEMS IN THE THEORY OF GRAPHS. III
• Correlation kernels for sums and products of random matrices
• Deep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy?
• Entropy and mutual information in models of deep neural networks*
• Freeness over the diagonal for large random matrices
• Identities from weighted Motzkin paths
• Kernel spectral clustering of large dimensional data
• Nonlinear random matrix theory for deep learning
• On the empirical distribution of eigenvalues of a class of large dimensional random matrices
• Products of many large random matrices and gradients in deep neural networks
• Raney distributions and random matrix theory
• Singular values of products of Ginibre random matrices, multiple orthogonal polynomials and hard edge scaling limits
• Spectral analysis of large dimensional random matrices
• Spectral analysis of the Gram matrix of mixture models
• The spectral norm of random inner-product kernel matrices
• The spectrum of kernel random matrices
• The spectrum of random inner-product kernel matrices
• Universal microscopic correlation functions for products of independent Ginibre matrices

Cited in

(9)

• Right large deviation principle for the top eigenvalue of the sum or product of invariant random matrices
• Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networks
• Structured random matrices and cyclic cumulants: a free probability approach
• A random matrix approach to neural networks
• Eigenvalue distribution of large random matrices arising in deep neural networks: Orthogonal case
• Large-dimensional random matrix theory and its applications in deep learning and wireless communications
• Nonlinear random matrix theory for deep learning
• On random matrices arising in deep neural networks: General I.I.D. case
• A note on the Pennington-Worah distribution

Describes a project that uses

Uses Software

• lsd
• dnner
• AlexNet
• GNMT
• ImageNet

This page was built for publication: Eigenvalue distribution of some nonlinear models of random matrices

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2076620)