A fully stochastic second-order trust region method
From MaRDI portal
Abstract: A stochastic second-order trust region method is proposed, which can be viewed as a second-order extension of the trust-region-ish (TRish) algorithm proposed by Curtis et al. (INFORMS J. Optim. 1(3) 200-220, 2019). In each iteration, a search direction is computed by (approximately) solving a trust region subproblem defined by stochastic gradient and Hessian estimates. The algorithm has convergence guarantees for stochastic minimization in the fully stochastic regime, meaning that guarantees hold when each stochastic gradient is required merely to be an unbiased estimate of the true gradient with bounded variance and when the stochastic Hessian estimates are bounded uniformly in norm. The algorithm is also equipped with a worst-case complexity guarantee in the nearly deterministic regime, i.e., when the stochastic gradient and Hessian estimates are very close in expectation to the true gradients and Hessians. The results of numerical experiments for training convolutional neural networks for image classification and training a recurrent neural network for time series forecasting are presented. These results show that the algorithm can outperform a stochastic gradient approach and the first-order TRish algorithm in practice.
Recommendations
- Stochastic trust-region methods with trust-region radius depending on probabilistic models
- A stochastic trust region method for unconstrained optimization problems
- Second-order stochastic optimization for machine learning in linear time
- A stochastic first-order trust-region method with inexact restoration for finite-sum minimization
- Stochastic quasi-Newton with line-search regularisation
Cites work
- A Stochastic Approximation Method
- A stochastic Levenberg-Marquardt method using random models with complexity results
- A stochastic Newton-Raphson method
- A stochastic quasi-Newton method for large-scale optimization
- Adaptive subgradient methods for online learning and stochastic optimization
- Complexity and global rates of trust-region methods based on probabilistic models
- Convergence of trust-region methods based on probabilistic models
- Exact and inexact subsampled Newton methods for optimization
- Exploiting negative curvature in deterministic and stochastic optimization
- Global convergence rate analysis of unconstrained optimization methods based on probabilistic models
- scientific article; zbMATH DE number 3449561 (Why is no real title available?)
- scientific article; zbMATH DE number 7306852 (Why is no real title available?)
- scientific article; zbMATH DE number 5060482 (Why is no real title available?)
- Hybrid deterministic-stochastic methods for data fitting
- Introductory lectures on convex optimization. A basic course.
- On a Stochastic Approximation Method
- Optimization methods for large-scale machine learning
- Probability
- Robust Stochastic Approximation Approach to Stochastic Programming
- Sample size selection in optimization methods for machine learning
- Stochastic block mirror descent methods for nonsmooth and stochastic optimization
- Stochastic First- and Zeroth-Order Methods for Nonconvex Stochastic Programming
- Stochastic optimization using a trust-region method and random models
- Stochastic Quasi-Newton Methods for Nonconvex Stochastic Optimization
- The Conjugate Gradient Method and Trust Regions in Large Scale Optimization
- Trust Region Methods
Cited in
(10)- A stochastic first-order trust-region method with inexact restoration for finite-sum minimization
- scientific article; zbMATH DE number 1186887 (Why is no real title available?)
- Stochastic trust-region methods with trust-region radius depending on probabilistic models
- scientific article; zbMATH DE number 5232303 (Why is no real title available?)
- LSOS: Line-search second-order stochastic optimization methods for nonconvex finite sums
- Fully stochastic trust-region sequential quadratic programming for equality-constrained optimization problems
- A non-monotone trust-region method with noisy oracles and additional sampling
- An investigation of stochastic trust-region based algorithms for finite-sum minimization
- Fully stochastic trust-region methods with Barzilai-Borwein steplengths
- An accelerated stochastic trust region method for stochastic optimization
This page was built for publication: A fully stochastic second-order trust region method
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5043844)
