Universal characteristics of deep neural network loss surfaces from random matrix theory

DOI10.1088/1751-8121/aca7f5OpenAlexW4311191227MaRDI QIDQ5878969

Francesco Mezzadri, Jonathan P. Keating, Diego Granziol, Nicholas P. Baskerville, Joseph Najnudel

Publication date: 23 February 2023

Published in: Journal of Physics A: Mathematical and Theoretical (Search for Journal in Brave)

Full work available at URL: https://arxiv.org/abs/2205.08601

zbMATH Keywords

neural networksuniversalityrandom matrix theoryloss surfaces

Mathematics Subject Classification ID

Computer science (68-XX) Statistical mechanics, structure of matter (82-XX)

Uses Software

• NetworkX
• PyTorch

Cites Work

• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• The eigenvector moment flow and local quantum unique ergodicity
• Characteristic polynomials for 1D random band matrices from the localization side
• The eigenvalues and eigenvectors of finite, low rank perturbations of large random matrices
• Anisotropic local laws for random matrices
• Universality for random matrix flows with time-dependent density
• Bulk universality for generalized Wigner matrices
• Eigenvectors distribution and quantum unique ergodicity for deformed Wigner matrices
• Optimal delocalization for generalized Wigner matrices
• A spin glass model for the loss surfaces of generative adversarial networks
• Characteristic polynomials for random band matrices near the threshold
• On the support of the free additive convolution
• Local Kesten-McKay law for random regular graphs
• Isotropic local laws for sample covariance and generalized Wigner matrices
• Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices
• Global fluctuations for 1D log-gas dynamics. Covariance kernel and support
• Complexity of Random Energy Landscapes, Glass Transition, and Absolute Value of the Spectral Determinant of Random Matrices
• The loss surfaces of neural networks with general activation functions
• The Lanczos and conjugate gradient algorithms in finite precision arithmetic
• An Introduction to Random Matrices
• RANDOM MATRICES WITH SLOW CORRELATION DECAY
• Generating Random Regular Graphs Quickly
• Universality of local spectral statistics of random matrices
• Random Matrices and Complexity of Spin Glasses
• On Random Matrices Arising in Deep Neural Networks. Gaussian Case
• Random Fields and Geometry
• A Dynamical Approach to Random Matrix Theory
• The space of interactions in neural network models
• Comparing dynamics: deep neural networks versus glassy systems
• Generating random regular graphs