A stochastic quasi-Newton method for large-scale optimization
From MaRDI portal
Abstract: The question of how to incorporate curvature information in stochastic approximation methods is challenging. The direct application of classical quasi- Newton updating techniques for deterministic optimization leads to noisy curvature estimates that have harmful effects on the robustness of the iteration. In this paper, we propose a stochastic quasi-Newton method that is efficient, robust and scalable. It employs the classical BFGS update formula in its limited memory form, and is based on the observation that it is beneficial to collect curvature information pointwise, and at regular intervals, through (sub-sampled) Hessian-vector products. This technique differs from the classical approach that would compute differences of gradients, and where controlling the quality of the curvature estimates can be difficult. We present numerical results on problems arising in machine learning that suggest that the proposed method shows much promise.
Recommendations
- On the use of stochastic Hessian information in optimization methods for machine learning
- Stochastic Quasi-Newton Methods for Nonconvex Stochastic Optimization
- Global convergence of online limited memory BFGS
- Stochastic quasi-Newton with line-search regularisation
- scientific article; zbMATH DE number 7353599
Cites work
- A coordinate gradient descent method for nonsmooth separable minimization
- A stochastic approximation algorithm with step-size adaptation
- A Stochastic Approximation Method
- Adaptive subgradient methods for online learning and stochastic optimization
- Convergence rate of incremental subgradient algorithms
- Global convergence of online limited memory BFGS
- scientific article; zbMATH DE number 3529352 (Why is no real title available?)
- scientific article; zbMATH DE number 1243473 (Why is no real title available?)
- scientific article; zbMATH DE number 1569104 (Why is no real title available?)
- Numerical Optimization
- On stochastic gradient and subgradient methods with adaptive steplength sequences
- On the use of stochastic Hessian information in optimization methods for machine learning
- RES: Regularized Stochastic BFGS Algorithm
- Robust Stochastic Approximation Approach to Stochastic Programming
- Sample size selection in optimization methods for machine learning
- SGD-QN: careful quasi-Newton stochastic gradient descent
- Stochastic simulation: Algorithms and analysis
Cited in
(only showing first 100 items - show all)- stochQN
- Adaptive stochastic approximation algorithm
- Global convergence of a modified two-parameter scaled BFGS method with Yuan-Wei-Lu line search for unconstrained optimization
- An information based approach to stochastic control problems
- A variation of Broyden class methods using Householder adaptive transforms
- Evolutionary prediction of nonstationary event popularity dynamics of Weibo social network using time-series characteristics
- Stochastic quasi-gradient methods: variance reduction via Jacobian sketching
- On the local convergence of a stochastic semismooth Newton method for nonsmooth nonconvex optimization
- SABRINA: a stochastic subspace majorization-minimization algorithm
- A subsampling approach for Bayesian model selection
- A hybrid stochastic optimization framework for composite nonconvex optimization
- Bayesian sparse learning with preconditioned stochastic gradient MCMC and its applications
- An adaptive Hessian approximated stochastic gradient MCMC method
- Limited-memory BFGS with displacement aggregation
- A stochastic extra-step quasi-Newton method for nonsmooth nonconvex optimization
- A new robust class of skew elliptical distributions
- Sketch-based empirical natural gradient methods for deep learning
- Two-dimensional distribution of streamwise velocity in open channel flow using maximum entropy principle: incorporation of additional constraints based on conservation laws
- Inexact restoration with subsampled trust-region methods for finite-sum minimization
- Use of projective coordinate descent in the Fekete problem
- Combining stochastic adaptive cubic regularization with negative curvature for nonconvex optimization
- Generalized self-concordant functions: a recipe for Newton-type methods
- A globally convergent incremental Newton method
- A framework for parallel second order incremental optimization algorithms for solving partially separable problems
- Stochastic quasi-Newton with line-search regularisation
- Secant penalized BFGS: a noise robust quasi-Newton method via penalizing the secant condition
- On the asymptotic rate of convergence of stochastic Newton algorithms and their weighted averaged versions
- Global convergence of online limited memory BFGS
- Newton Sketch: A Near Linear-Time Optimization Algorithm with Linear-Quadratic Convergence
- On the use of stochastic Hessian information in optimization methods for machine learning
- Going Off the Grid: Iterative Model Selection for Biclustered Matrix Completion
- Weighted SGD for _p regression with randomized preconditioning
- Randomized quasi-Newton updates are linearly convergent matrix inversion algorithms
- Utilizing second order information in minibatch stochastic variance reduced proximal iterations
- Second-order stochastic optimization for machine learning in linear time
- Block BFGS methods
- scientific article; zbMATH DE number 6870925 (Why is no real title available?)
- Quasi-Newton methods: superlinear convergence without line searches for self-concordant functions
- Efficient computation of derivatives for solving optimization problems in R and Python using SWIG-generated interfaces to ADOL-C
- scientific article; zbMATH DE number 4118164 (Why is no real title available?)
- A robust multi-batch L-BFGS method for machine learning
- Sampled Tikhonov regularization for large linear inverse problems
- Sampled limited memory methods for massive linear inverse problems
- A noise-tolerant quasi-Newton algorithm for unconstrained optimization
- A fully stochastic second-order trust region method
- scientific article; zbMATH DE number 7626758 (Why is no real title available?)
- Quasi-Newton methods for machine learning: forget the past, just sample
- A Stochastic Second-Order Generalized Estimating Equations Approach for Estimating Association Parameters
- QNG: a quasi-natural gradient method for large-scale statistical learning
- Reduced approach for stochastic optimal control problems
- A Variable Sample-Size Stochastic Quasi-Newton Method for Smooth and Nonsmooth Stochastic Convex Optimization
- A New Likelihood Ratio Method for Training Artificial Neural Networks
- slimTrain---A Stochastic Approximation Method for Training Separable Deep Neural Networks
- Large Sample Mean-Field Stochastic Optimization
- On stochastic and deterministic quasi-Newton methods for nonstrongly convex optimization: asymptotic convergence and rate analysis
- Trust-region algorithms for training responses: machine learning methods using indefinite Hessian approximations
- Convergence of inexact forward-backward algorithms using the forward-backward envelope
- An inertial Newton algorithm for deep learning
- Are we forgetting about compositional optimisers in Bayesian optimisation?
- Stochastic proximal quasi-Newton methods for non-convex composite optimization
- Stochastic sub-sampled Newton method with variance reduction
- Analysis of the BFGS Method with Errors
- An efficient stochastic Newton algorithm for parameter estimation in logistic regressions
- An inexact variable metric proximal point algorithm for generic quasi-Newton acceleration
- Quasi-Newton smoothed functional algorithms for unconstrained and constrained simulation optimization
- A stochastic semismooth Newton method for nonsmooth nonconvex optimization
- Newton-like method with diagonal correction for distributed optimization
- Kalman-based stochastic gradient method with stop condition and insensitivity to conditioning
- Stochastic Quasi-Newton Methods for Nonconvex Stochastic Optimization
- IQN: an incremental quasi-Newton method with local superlinear convergence rate
- LSOS: Line-search second-order stochastic optimization methods for nonconvex finite sums
- Towards explicit superlinear convergence rate for SR1
- Efficient learning rate adaptation based on hierarchical optimization approach
- A stochastic variance reduced gradient using Barzilai-Borwein techniques as second order information
- An overview of stochastic quasi-Newton methods for large-scale machine learning
- Two-stage 2D-to-3d reconstruction of realistic microstructures: implementation and numerical validation by effective properties
- Adaptive step size rules for stochastic optimization in large-scale learning
- A modified stochastic quasi-Newton algorithm for summing functions problem in machine learning
- On the complexity of a stochastic Levenberg-Marquardt method
- The regularized stochastic Nesterov's accelerated quasi-Newton method with applications
- On pseudoinverse-free block maximum residual nonlinear Kaczmarz method for solving large-scale nonlinear system of equations
- Riemannian Natural Gradient Methods
- A single timescale stochastic quasi-Newton method for stochastic optimization
- A branch-and-bound algorithm with growing datasets for large-scale parameter estimation
- Incremental quasi-Newton algorithms for solving a nonconvex, nonsmooth, finite-sum optimization problem
- Resource-adaptive Newton's method for distributed learning
- Subsampled first-order optimization methods with applications in imaging
- Stochastic Steffensen method
- On the inversion-free Newton's method and its applications
- An efficient averaged stochastic Gauss-Newton algorithm for estimating parameters of nonlinear regressions models
- Bolstering stochastic gradient descent with model building
- A proximal stochastic quasi-Newton algorithm with dynamical sampling and stochastic line search
- Adaptive pruning-based Newton's method for distributed learning
- Approximating Hessian matrices using Bayesian inference: a new approach for quasi-Newton methods in stochastic optimization
- Spectral projected gradient method for stochastic optimization
- Adaptive pseudo-solution-assisted PINNs for multiscale Darcy and Darcy-Forchheimer equations
- Physics-informed neural networks for high-dimensional solutions and snaking bifurcations in nonlinear lattices
- Sequential sample average majorization-minimization
- An efficient low-complexity stochastic BFGS algorithm using matrix diagonal approximations for nonconvex optimization in machine learning
- Online estimation and inference for robust policy evaluation in reinforcement learning
This page was built for publication: A stochastic quasi-Newton method for large-scale optimization
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q121136)
