A rigorous framework for the mean field limit of multilayer neural networks
type: Item
instance of: scholarly article
title: A rigorous framework for the mean field limit of multilayer neural networks
author: Phan Nguyen
author: Huy-Tuan Pham
published in: Mathematical Statistics and Learning
publication date: 2023-11-06T00:00:00Z
full work available at URL: https://arxiv.org/abs/2001.11443
review text: Summary: We develop a mathematically rigorous framework for multilayer neural networks in the mean field regime. As the network's widths increase, the network's learning trajectory is shown to be well captured by a meaningful and dynamically nonlinear limit (the \textit{mean field} limit), which is characterized by a system of ODEs. Our framework applies to a broad range of network architectures, learning dynamics and network initializations. Central to the framework is the new idea of a \textit{neuronal embedding}, which comprises of a non-evolving probability space that allows to embed neural networks of arbitrary widths. Using our framework, we prove several properties of large-width multilayer neural networks. Firstly we show that independent and identically distributed initializations cause strong degeneracy effects on the network's learning trajectory when the network's depth is at least four. Secondly we obtain several global convergence guarantees for feedforward multilayer networks under a number of different setups. These include two-layer and three-layer networks with independent and identically distributed initializations, and multilayer networks of arbitrary depths with a special type of correlated initializations that is motivated by the new concept of \textit{bidirectional diversity}. Unlike previous works that rely on convexity, our results admit non-convex losses and hinge on a certain universal approximation property, which is a distinctive feature of infinite-width neural networks and it is shown to hold throughout the training process. Aside from being the first known results for global convergence of multilayer networks in the mean field regime, they demonstrate flexibility of our framework and incorporate several new ideas and insights that depart from the conventional convex optimization wisdom.
Mathematics Subject Classification ID: 62G08
Mathematics Subject Classification ID: 82C22
Mathematics Subject Classification ID: 90C26
Mathematics Subject Classification ID: 68T07
zbMATH DE Number: 7761504
zbMATH Keywords: neural network
zbMATH Keywords: deep learning
zbMATH Keywords: mean field limit
zbMATH Keywords: optimization
zbMATH Keywords: global convergence
MaRDI profile type: MaRDI publication profile
cites work: Gradient flows in metric spaces and in the space of probability measures
cites work: Sparse optimization on measures with over-parameterized gradient descent
cites work: Approximation by superpositions of a sigmoidal function
cites work: Machine learning from a continuous viewpoint. I
cites work: Analysis of a two-layer neural network via displacement convexity
cites work: Wide neural networks of any depth evolve as linear models under gradient descent *
cites work: A mean field view of the landscape of two-layer neural networks
cites work: Particle dual averaging: optimization of mean field neural network with global convergence rate analysis*
cites work: Mean field analysis of neural networks: a law of large numbers
cites work: Mean Field Analysis of Deep Neural Networks
cites work: Q3358031
cites work: Gradient descent optimizes over-parameterized deep ReLU networks
DOI: 10.4171/MSL/42
description: scientific article; zbMATH DE number 7761504