Scheduled restart momentum for accelerated stochastic gradient descent
From MaRDI portal
(Redirected from Publication:5094616)
Abstract: Stochastic gradient descent (SGD) with constant momentum and its variants such as Adam are the optimization algorithms of choice for training deep neural networks (DNNs). Since DNN training is incredibly computationally expensive, there is great interest in speeding up the convergence. Nesterov accelerated gradient (NAG) improves the convergence rate of gradient descent (GD) for convex optimization using a specially designed momentum; however, it accumulates error when an inexact gradient is used (such as in SGD), slowing convergence at best and diverging at worst. In this paper, we propose Scheduled Restart SGD (SRSGD), a new NAG-style scheme for training DNNs. SRSGD replaces the constant momentum in SGD by the increasing momentum in NAG but stabilizes the iterations by resetting the momentum to zero according to a schedule. Using a variety of models and benchmarks for image classification, we demonstrate that, in training DNNs, SRSGD significantly improves convergence and generalization; for instance in training ResNet200 for ImageNet classification, SRSGD achieves an error rate of 20.93% vs. the benchmark of 22.13%. These improvements become more significant as the network grows deeper. Furthermore, on both CIFAR and ImageNet, SRSGD reaches similar or even better error rates with significantly fewer training epochs compared to the SGD baseline.
Recommendations
- Stochastic gradient descent with Polyak's learning rate
- Adaptive restart for accelerated gradient schemes
- Katyusha: the first direct acceleration of stochastic gradient methods
- Katyusha: the first direct acceleration of stochastic gradient methods
- An adaptive gradient method with energy and momentum
Cites work
- A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems
- Accelerated gradient methods for nonconvex nonlinear and stochastic programming
- Adaptive restart for accelerated gradient schemes
- Adaptive subgradient methods for online learning and stochastic optimization
- An optimal method for stochastic composite optimization
- Convex Analysis
- First-order methods of smooth convex optimization with inexact oracle
- Gradient methods for minimizing composite functions
- scientific article; zbMATH DE number 3850830 (Why is no real title available?)
- Introductory lectures on convex optimization. A basic course.
- Optimal methods of smooth convex minimization
- Optimization methods for large-scale machine learning
- Robust accelerated gradient methods for smooth strongly convex functions
- Some methods of speeding up the convergence of iteration methods
- Stochastic First- and Zeroth-Order Methods for Nonconvex Stochastic Programming
Cited in
(6)- How does momentum benefit deep neural networks architecture design? A few case studies
- Accelerating deep neural network training with inconsistent stochastic gradient descent
- Adaptive and Implicit Regularization for Matrix Completion
- Learning proper orthogonal decomposition of complex dynamics using heavy-ball neural ODEs
- Decentralized concurrent learning with coordinated momentum and restart
- A novel stochastic conjugate gradient algorithm based on a stochastic differential equation perspective
Describes a project that uses
Uses Software
This page was built for publication: Scheduled restart momentum for accelerated stochastic gradient descent
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5094616)
