On random matrices arising in deep neural networks: General I.I.D. case

DOI10.1142/S2010326322500460MaRDI QIDQ6163573zbMATH OpenOpenAlexWikidataFDO

Authors L. A. Pastur, Victor Slavin

Publication date 26 June 2023

Published in Random Matrices: Theory and Applications (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2011.11439

zbMATH Keywords

eigenvalue distributionrandom matricesdeep neural networks

Mathematics Subject Classification ID

Random matrices (probabilistic aspects) (60B20) Neural networks for/in biological studies, artificial life and related topics (92B20)

Abstract: We study the distribution of singular values of product of random matrices pertinent to the analysis of deep neural networks. The matrices resemble the product of the sample covariance matrices, however, an important difference is that the population covariance matrices assumed to be non-random or random but independent of the random data matrix in statistics and random matrix theory are now certain functions of random data matrices (synaptic weight matrices in the deep neural network terminology). The problem has been treated in recent work [25, 13] by using the techniques of free probability theory. Since, however, free probability theory deals with population covariance matrices which are independent of the data matrices, its applicability has to be justified. The justification has been given in [22] for Gaussian data matrices with independent entries, a standard analytical model of free probability, by using a version of the techniques of random matrix theory. In this paper we use another, more streamlined, version of the techniques of random matrix theory to generalize the results of [22] to the case where the entries of the synaptic weight matrices are just independent identically distributed random variables with zero mean and finite fourth moment. This, in particular, extends the property of the so-called macroscopic universality on the considered random matrices.

Recommendations

• On Random Matrices Arising in Deep Neural Networks. Gaussian Case
• Eigenvalue distribution of large random matrices arising in deep neural networks: Orthogonal case
• scientific article; zbMATH DE number 822011
• Nonlinear random matrix theory for deep learning
• Eigenvalue distribution of some nonlinear models of random matrices

Cites work

• scientific article; zbMATH DE number 1722641 (Why is no real title available?)
• scientific article; zbMATH DE number 5943539 (Why is no real title available?)
• scientific article; zbMATH DE number 6125590 (Why is no real title available?)
• Analysis of the limiting spectral measure of large random matrices of the separable covariance type
• Asymptotic spectra of matrix-valued functions of independent random matrices and free probability
• DISTRIBUTION OF EIGENVALUES FOR SOME SETS OF RANDOM MATRICES
• Deep Neural Networks in a Mathematical Framework
• Deep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy?
• Deep learning
• Eigenvalue distribution of large random matrices
• High-dimensional probability. An introduction with applications in data science
• On the asymptotic eigenvalue distribution of concatenated vector-valued fading channels
• Random matrices: universality of ESDs and the circular law
• Spectral analysis of large dimensional random matrices
• The Principles of Deep Learning Theory
• The loss surfaces of neural networks with general activation functions

Cited in

(13)

• The Law of Multiplication of Large Random Matrices Revisited
• Large-dimensional random matrix theory and its applications in deep learning and wireless communications
• Products of many large random matrices and gradients in deep neural networks
• Linear eigenvalue statistics of XX′ matrices
• Nonlinear random matrix theory for deep learning
• A random matrix approach to neural networks
• Eigenvalue distribution of some nonlinear models of random matrices
• A note on the Pennington-Worah distribution
• Asymptotic freeness of layerwise Jacobians caused by invariance of multilayer perceptron: the Haar orthogonal case
• Eigenvalue distribution of large random matrices arising in deep neural networks: Orthogonal case
• Universal characteristics of deep neural network loss surfaces from random matrix theory
• On Random Matrices Arising in Deep Neural Networks. Gaussian Case
• scientific article; zbMATH DE number 7415108 (Why is no real title available?)

This page was built for publication: On random matrices arising in deep neural networks: General I.I.D. case

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6163573)