Gradient descent on infinitely wide neural networks: global convergence and generalization

DOI10.4171/ICM2022/121arXiv2110.08084MaRDI QIDQ6200217FDOQ6200217

Lénaïc Chizat, Francis Bach

Publication date: 22 March 2024

Published in: International Congress of Mathematicians (Search for Journal in Brave)

Abstract: Many supervised machine learning methods are naturally cast as optimization problems. For prediction models which are linear in their parameters, this often leads to convex problems for which many mathematical guarantees exist. Models which are non-linear in their parameters such as neural networks lead to non-convex optimization problems for which guarantees are harder to obtain. In this review paper, we consider two-layer neural networks with homogeneous activation functions where the number of hidden neurons tends to infinity, and show how qualitative convergence guarantees may be derived.

Full work available at URL: https://arxiv.org/abs/2110.08084

zbMATH Keywords

machine learningoptimal transportgradient flowneural networksgradient descent

Mathematics Subject Classification ID

Numerical optimization and variational techniques (65K10) Artificial neural networks and deep learning (68T07) Numerical methods based on nonlinear programming (49M37) Methods of reduced gradient type (90C52) Probabilistic metric spaces (54E70) Probabilistic methods in Banach space theory (46B09)

Cites Work

• Universal approximation bounds for superpositions of a sigmoidal function
• Optimal transport for applied mathematicians. Calculus of variations, PDEs, and modeling
• Title not available (Why is that?)
• Asymptotic Statistics
• Gradient flows in metric spaces and in the space of probability measures
• Title not available (Why is that?)
• Empirical margin distributions and bounding the generalization error of combined classifiers
• An Introduction to Numerical Analysis
• Convex optimization: algorithms and complexity
• Probabilistic representation and uniqueness results for measure-valued solutions of transport equations
• Title not available (Why is that?)
• Bounds on rates of variable-basis and neural-network approximation
• Lectures on convex optimization
• A mean field view of the landscape of two-layer neural networks
• Breaking the Curse of Dimensionality with Convex Neural Networks
• Title not available (Why is that?)
• Title not available (Why is that?)
• Mean Field Analysis of Neural Networks: A Law of Large Numbers

Cited In (2)

• Infinite-width limit of deep linear neural networks
• Numerical solution of Poisson partial differential equation in high dimension using two-layer neural networks

This page was built for publication: Gradient descent on infinitely wide neural networks: global convergence and generalization

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6200217)