Mean field analysis of neural networks: a law of large numbers
From MaRDI portal
Publication:5219306
DOI10.1137/18M1192184zbMATH Open1440.60008arXiv1805.01053OpenAlexW3010825589WikidataQ114847156 ScholiaQ114847156MaRDI QIDQ5219306FDOQ5219306
Justin A. Sirignano, Konstantinos Spiliopoulos
Publication date: 11 March 2020
Published in: SIAM Journal on Applied Mathematics (Search for Journal in Brave)
Abstract: Machine learning, and in particular neural network models, have revolutionized fields such as image, text, and speech recognition. Today, many important real-world applications in these areas are driven by neural networks. There are also growing applications in engineering, robotics, medicine, and finance. Despite their immense success in practice, there is limited mathematical understanding of neural networks. This paper illustrates how neural networks can be studied via stochastic analysis, and develops approaches for addressing some of the technical challenges which arise. We analyze one-layer neural networks in the asymptotic regime of simultaneously (A) large network sizes and (B) large numbers of stochastic gradient descent training iterations. We rigorously prove that the empirical distribution of the neural network parameters converges to the solution of a nonlinear partial differential equation. This result can be considered a law of large numbers for neural networks. In addition, a consequence of our analysis is that the trained parameters of the neural network asymptotically become independent, a property which is commonly called "propagation of chaos".
Full work available at URL: https://arxiv.org/abs/1805.01053
Recommendations
- Mean field analysis of neural networks: a central limit theorem
- Mean Field Analysis of Deep Neural Networks
- Asymptotic properties of one-layer artificial neural networks with sparse connectivity
- Statistical guarantees for regularized neural networks
- A mean field view of the landscape of two-layer neural networks
Computational methods for problems pertaining to probability theory (60-08) Neural nets and related approaches to inference from stochastic processes (62M45) Strong limit theorems (60F15)
Cites Work
- DGM: a deep learning algorithm for solving partial differential equations
- Gradient flows in metric spaces and in the space of probability measures
- Title not available (Why is that?)
- Kinetic equilibration rates for granular media and related equations: entropy dissipation and mass transportation estimates
- Title not available (Why is that?)
- Title not available (Why is that?)
- Title not available (Why is that?)
- The Variational Formulation of the Fokker--Planck Equation
- Multilayer feedforward networks are universal approximators
- Default clustering in large portfolios: typical events
- McKean-Vlasov limit for interacting random processes in random media.
- Separability and completeness for the Wasserstein distance
- Large deviations and mean-field theory for asymmetric random recurrent neural networks
- Large portfolio losses: A dynamic contagion model
- Mean-field Langevin dynamics and energy landscape of neural networks
- LARGE PORTFOLIO ASYMPTOTICS FOR LOSS FROM DEFAULT
- Heterogeneous credit portfolios and the dynamics of the aggregate losses
- Approximation and estimation bounds for artificial neural networks
- Particle systems with a singular mean-field self-excitation. Application to neuronal networks
- Mean-Field Limit of a Stochastic Particle System Smoothly Interacting Through Threshold Hitting-Times and Applications to Neural Networks with Dendritic Component
- Machine learning strategies for systems with invariance properties
- Reynolds averaged turbulence modelling using deep neural networks with embedded invariance
- A stochastic McKean-Vlasov equation for absorbing diffusions on the half-line
- Mean field analysis of neural networks: a central limit theorem
- A mean field view of the landscape of two-layer neural networks
- Systemic Risk in Interbanking Networks
- Universal features of price formation in financial markets: perspectives from deep learning
Cited In (46)
- Asymptotics of Reinforcement Learning with Neural Networks
- Continuous limits of residual neural networks in case of large input data
- Suboptimal Local Minima Exist for Wide Neural Networks with Smooth Activations
- Asymptotic properties of one-layer artificial neural networks with sparse connectivity
- Large Sample Mean-Field Stochastic Optimization
- Two-Layer Neural Networks with Values in a Banach Space
- Mean Field Analysis of Deep Neural Networks
- Representation formulas and pointwise properties for Barron functions
- Optimization in machine learning: a distribution-space approach
- Sharp uniform-in-time propagation of chaos
- Mean-field Langevin dynamics and energy landscape of neural networks
- Infinite-width limit of deep linear neural networks
- A unified Fourier slice method to derive ridgelet transform for a variety of depth-2 neural networks
- Reinforcement learning and stochastic optimisation
- A rigorous framework for the mean field limit of multilayer neural networks
- Mean-field inference methods for neural networks
- Online parameter estimation for the McKean-Vlasov stochastic differential equation
- Mehler’s Formula, Branching Process, and Compositional Kernels of Deep Neural Networks
- Consensus-based optimization methods converge globally
- The Continuous Formulation of Shallow Neural Networks as Wasserstein-Type Gradient Flows
- Learning sparse features can lead to overfitting in neural networks
- Title not available (Why is that?)
- A selective overview of deep learning
- Stochastic gradient descent with noise of machine learning type. II: Continuous time analysis
- A class of dimension-free metrics for the convergence of empirical measures
- Statistical guarantees for regularized neural networks
- Fast Non-mean-field Networks: Uniform in Time Averaging
- Non-mean-field Vicsek-type models for collective behavior
- Normalization effects on deep neural networks
- Gradient descent on infinitely wide neural networks: global convergence and generalization
- Large deviations for nonlocal stochastic neural fields
- Markov chain network training and conservation law approximations: Linking microscopic and macroscopic models for evolution
- Landscape and training regimes in deep learning
- Deep learning: a statistical viewpoint
- Large deviation analysis of function sensitivity in random deep neural networks
- Title not available (Why is that?)
- Normalization effects on shallow neural networks and related asymptotic expansions
- Supervised learning from noisy observations: combining machine-learning techniques with data assimilation
- Title not available (Why is that?)
- Propagation of chaos: a review of models, methods and applications. I: Models and methods
- Sparse optimization on measures with over-parameterized gradient descent
- A blob method for inhomogeneous diffusion with applications to multi-agent control and sampling
- Propagation of chaos: a review of models, methods and applications. II: Applications
- Nonlocal cross-diffusion systems for multi-species populations and networks
- Surprises in high-dimensional ridgeless least squares interpolation
- Mean Field Limits for Interacting Diffusions with Colored Noise: Phase Transitions and Spectral Numerical Methods
Uses Software
This page was built for publication: Mean field analysis of neural networks: a law of large numbers
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5219306)
