Mini-batch descent in semiflows
Evolution inclusions (34G25) Nonlinear parabolic equations (35K55) Dynamics induced by flows and semiflows (37C10) General theory of infinite-dimensional dissipative dynamical systems, nonlinear semigroups, evolution equations (37L05) Nonsmooth analysis (49J52) Numerical solutions to abstract evolution equations (65J08)
This paper considers applying mini-batch gradient descent to semiflows. The authors study the gradient flow problem\N\[\N\dot{u}(t)\in-\partial\Phi(t),\N\] \N\[\Nu(0)=u_{0}\in \text{dom}\partial\Phi\N\]\Nwhere \(\Phi:H\rightarrow\mathbb{R}\cup\{\infty\}\) is a convex, lower semicontinuous, proper function and \(H\) is a Hilbert space. It is assumed that \(\Phi\) can be represented as\N\[\N\Phi=\sum_{i=1}^{n} p_{i}\Phi_{i}\text{, where }\sum_{i=1}^{n} p_{i}=1.\N\]\NA variation of stochastic descent algorithms is mini-batch gradient descent. The authors compare the gradient flow of the full loss function with a flow that does not strictly follow the steepest descent of the full loss function but instead follows portions of it, determined by randomly selected batches of sub-loss functions changing over time, thus defining a piecewise flow. If the replacement of batches is done sufficiently often, the authors show that the trajectory generated by the mini-batch descent flow is close in expectation to the trajectory generated by the gradient flow of the full loss function. They also define a randomized version of the minimizing movement scheme and consider its relationship with the gradient flow of the full loss function. They provide convergence estimates and conditions under which this scheme approximates the gradient flow.\N\NTheir mini-batch descent scheme is based on [\textit{S. Jin} et al., J. Comput. Phys. 400, Article ID 108877, 30 p. (2020; Zbl 1453.82065)]. This algorithm was also used in [\textit{D. W. M. Veldman} and \textit{E. Zuazua}, Numer. Math. 151, No. 2, 495--549 (2022; Zbl 1493.49035)] to generate approximate optimal control trajectories. The authors extend the convergence results for trajectories obtained there from positive definite matrices to convex functions and from Euclidean to Hilbert spaces.\N\NFinally, the authors' randomized schemes are applied to several problems and include numerical results: projected dynamics from constrained optimization, a sparse inversion problem and a random domain decomposition algorithm for parabolic-type partial differential equations.
- A dual condition for the convex subdifferential sum formula with applications
- A framework for randomized time-splitting in linear-quadratic optimal control
- A randomized operator splitting scheme inspired by stochastic optimization methods
- A smooth penalty approach and a nonlinear multigrid algorithm for elliptic MPECs
- Analysis of stochastic gradient descent in continuous time
- Convergence of the random batch method for interacting particles with disparate species and weights
- Convex analysis and monotone operator theory in Hilbert spaces
- Distribution Theory Applied to Differential Equations
- Equation of state calculations by fast computing machines
- Gradient flows and randomised thresholding: sparse inversion and classification
- scientific article; zbMATH DE number 3497315 (Why is no real title available?)
- scientific article; zbMATH DE number 5233919 (Why is no real title available?)
- Large-scale machine learning with stochastic gradient descent
- Model predictive control with random batch methods for a guiding problem
- Nonlinear Differential Equations of Monotone Types in Banach Spaces
- On the continuity of the time derivative of the solution to the parabolic obstacle problem with variable coefficients
- Optimal control of the obstacle for a parabolic variational inequality
- Optimization by simulated annealing
- Optimization methods for large-scale machine learning
- Parabolic variational inequalities in one space dimension and smoothness of the free boundary
- Random batch methods (RBM) for interacting particle systems
- Stability and convergence of a randomized model predictive control strategy
- The Monte Carlo Method
- Uniform error estimates for the random batch method to the first-order consensus models with antisymmetric interaction kernels
- Variational Analysis in Sobolev and BV Spaces
This page was built for publication: Mini-batch descent in semiflows
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6996843)
