Adaptivity of stochastic gradient methods for nonconvex optimization
From MaRDI portal
Abstract: Adaptivity is an important yet under-studied property in modern optimization theory. The gap between the state-of-the-art theory and the current practice is striking in that algorithms with desirable theoretical guarantees typically involve drastically different settings of hyperparameters, such as step-size schemes and batch sizes, in different regimes. Despite the appealing theoretical results, such divisive strategies provide little, if any, insight to practitioners to select algorithms that work broadly without tweaking the hyperparameters. In this work, blending the "geometrization" technique introduced by Lei & Jordan 2016 and the exttt{SARAH} algorithm of Nguyen et al., 2017, we propose the Geometrized exttt{SARAH} algorithm for non-convex finite-sum and stochastic optimization. Our algorithm is proved to achieve adaptivity to both the magnitude of the target accuracy and the Polyak-L{}ojasiewicz (PL) constant if present. In addition, it achieves the best-available convergence rate for non-PL objectives simultaneously while outperforming existing algorithms for PL objectives.
Recommendations
Cites work
- Accelerate stochastic subgradient method by leveraging local growth condition
- Acceleration of Stochastic Approximation by Averaging
- Adaptive subgradient methods for online learning and stochastic optimization
- Distributed optimization with arbitrary local solvers
- Fastest rates for stochastic mirror descent methods
- Harder, Better, Faster, Stronger Convergence Rates for Least-Squares Regression
- scientific article; zbMATH DE number 3790208 (Why is no real title available?)
- New method of stochastic approximation type
- On the adaptivity of stochastic gradient-based optimization
- Robust Stochastic Approximation Approach to Stochastic Programming
- Semi-stochastic coordinate descent
- Some methods of speeding up the convergence of iteration methods
- Stochastic dual coordinate ascent methods for regularized loss minimization
- Stochastic quasi-gradient methods: variance reduction via Jacobian sketching
- Universal gradient methods for convex optimization problems
Cited in
(11)- Contravariant adaptation on structured matrix spaces
- On the adaptivity of stochastic gradient-based optimization
- scientific article; zbMATH DE number 7306906 (Why is no real title available?)
- Adaptive FISTA for Nonconvex Optimization
- Adaptive Regularization Algorithms with Inexact Evaluations for Nonconvex Optimization
- Stochastic momentum methods for non-convex learning without bounded assumptions
- Byzantine-robust loopless stochastic variance-reduced gradient
- Taming Neural Networks with TUSLA: Nonconvex Learning via Adaptive Stochastic Gradient Langevin Algorithms
- Stochastic variable metric proximal gradient with variance reduction for non-convex composite optimization
- Improving the stochastically controlled stochastic gradient method by the bandwidth-based stepsize
- Optimal complexity in Byzantine-robust distributed stochastic optimization with data heterogeneity
This page was built for publication: Adaptivity of stochastic gradient methods for nonconvex optimization
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5076671)
