Stochastic Modified Flows, Mean-Field Limits and Dynamics of Stochastic Gradient Descent
From MaRDI portal
Publication:6426436
arXiv2302.07125MaRDI QIDQ6426436FDOQ6426436
Authors: Benjamn Gess, Sebastian Kassing, Vitalii Konarovskyi 
Publication date: 14 February 2023
Abstract: We propose new limiting dynamics for stochastic gradient descent in the small learning rate regime called stochastic modified flows. These SDEs are driven by a cylindrical Brownian motion and improve the so-called stochastic modified equations by having regular diffusion coefficients and by matching the multi-point statistics. As a second contribution, we introduce distribution dependent stochastic modified flows which we prove to describe the fluctuating limiting dynamics of stochastic gradient descent in the small learning rate - infinite width scaling regime.
Stochastic partial differential equations (aspects of stochastic analysis) (60H15) Artificial neural networks and deep learning (68T07) Discrete-time Markov processes on general state spaces (60J05) Random measures (60G57) Martingales and classical analysis (60G46) Derivatives of functions in infinite-dimensional spaces (46G05)
This page was built for publication: Stochastic Modified Flows, Mean-Field Limits and Dynamics of Stochastic Gradient Descent
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6426436)
