Stochastic gradient descent with noise of machine learning type. II: Continuous time analysis

DOI10.1007/S00332-023-09992-0arXiv2106.02588OpenAlexW3166627127MaRDI QIDQ6188971FDOQ6188971

Stephan Wojtowytsch

Publication date: 12 January 2024

Published in: Journal of Nonlinear Science (Search for Journal in Brave)

Abstract: The representation of functions by artificial neural networks depends on a large number of parameters in a non-linear fashion. Suitable parameters of these are found by minimizing a 'loss functional', typically by stochastic gradient descent (SGD) or an advanced SGD-based algorithm. In a continuous time model for SGD with noise that follows the 'machine learning scaling', we show that in a certain noise regime, the optimization algorithm prefers 'flat' minima of the objective function in a sense which is different from the flat minimum selection of continuous time SGD with homogeneous noise.

Full work available at URL: https://arxiv.org/abs/2106.02588

zbMATH Keywords

deep learningmachine learningnonconvex optimizationstochastic differential equationstochastic gradient descentoverparametrizationinvariant distributiondegenerate diffusion equationimplicit biasglobal minimum selectionflat minimum selectionPoincaré-Hardy inequality

Mathematics Subject Classification ID

Artificial neural networks and deep learning (68T07) Nonconvex programming, global optimization (90C26) Degenerate parabolic equations (35K65) Applications of stochastic analysis (to PDEs, etc.) (60H30)

Cites Work

• A Stochastic Approximation Method
• Title not available (Why is that?)
• Elliptic partial differential equations of second order
• Functional analysis, Sobolev spaces and partial differential equations
• Regularity theory for general stable operators
• The Variational Formulation of the Fokker--Planck Equation
• Title not available (Why is that?)
• Regularity theory for general stable operators: parabolic equations
• Sharp rates of decay of solutions to the nonlinear fast diffusion equation via functional inequalities
• Bounds for the Discrete Part of the Spectrum of a Semi-Bounded Schrödinger Operator.
• Rectifiable sets, densities and tangent measures
• A Liouville Theorem for Degenerate Elliptic Equations
• Wahrscheinlichkeitstheorie
• Mean-field Langevin dynamics and energy landscape of neural networks
• Improved Poincaré inequalities
• Optimal control of stochastic differential equations via Fokker-Planck equations
• A Comprehensive Introduction to Sub-Riemannian Geometry
• Sub-Laplacian eigenvalue bounds on sub-Riemannian manifolds
• A mean field view of the landscape of two-layer neural networks
• Analysis of stochastic gradient descent in continuous time
• About the Hardy Inequality
• On the Heat Diffusion for Generic Riemannian and Sub-Riemannian Structures
• Stochastic Gradient Descent in Continuous Time
• Stochastic Gradient Descent in Continuous Time: A Central Limit Theorem
• Analysis of a two-layer neural network via displacement convexity
• Mean Field Analysis of Neural Networks: A Law of Large Numbers
• Stochastic gradient descent with noise of machine learning type. I: Discrete time analysis

Cited In (1)

• Stochastic modified flows for Riemannian stochastic gradient descent

This page was built for publication: Stochastic gradient descent with noise of machine learning type. II: Continuous time analysis

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6188971)