Stochastic L-BFGS: Improved Convergence Rates and Practical Acceleration Strategies
From MaRDI portal
Abstract: We revisit the stochastic limited-memory BFGS (L-BFGS) algorithm. By proposing a new framework for the convergence analysis, we prove improved convergence rates and computational complexities of the stochastic L-BFGS algorithms compared to previous works. In addition, we propose several practical acceleration strategies to speed up the empirical performance of such algorithms. We also provide theoretical analyses for most of the strategies. Experiments on large-scale logistic and ridge regression problems demonstrate that our proposed strategies yield significant improvements vis-`a-vis competing state-of-the-art algorithms.
Cited in
(8)- A stochastic extra-step quasi-Newton method for nonsmooth nonconvex optimization
- Secant penalized BFGS: a noise robust quasi-Newton method via penalizing the secant condition
- A machine-learning-accelerated distributed LBFGS method for field development optimization: algorithm, validation, and applications
- An overview of stochastic quasi-Newton methods for large-scale machine learning
- Incremental quasi-Newton algorithms for solving a nonconvex, nonsmooth, finite-sum optimization problem
- Stochastic Steffensen method
- An efficient low-complexity stochastic BFGS algorithm using matrix diagonal approximations for nonconvex optimization in machine learning
- Stochastic limited memory BFGS algorithms for solving nonconvex support vector machine problems and nonconvex risk minimization program
This page was built for publication: Stochastic L-BFGS: Improved Convergence Rates and Practical Acceleration Strategies
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q4621626)
