A robust multi-batch L-BFGS method for machine learning
From MaRDI portal
Abstract: This paper describes an implementation of the L-BFGS method designed to deal with two adversarial situations. The first occurs in distributed computing environments where some of the computational nodes devoted to the evaluation of the function and gradient are unable to return results on time. A similar challenge occurs in a multi-batch approach in which the data points used to compute function and gradients are purposely changed at each iteration to accelerate the learning process. Difficulties arise because L-BFGS employs gradient differences to update the Hessian approximations, and when these gradients are computed using different data points the updating process can be unstable. This paper shows how to perform stable quasi-Newton updating in the multi-batch setting, studies the convergence properties for both convex and nonconvex functions, and illustrates the behavior of the algorithm in a distributed computing platform on binary classification logistic regression and neural network training problems that arise in machine learning.
Recommendations
- A quasi-Newton approach to nonsmooth convex optimization problems in machine learning
- L-Broyden methods: a generalization of the L-BFGS method to the limited-memory Broyden family
- A noise-tolerant quasi-Newton algorithm for unconstrained optimization
- Global convergence of online limited memory BFGS
- scientific article; zbMATH DE number 7353599
Cites work
- A Family of Variable-Metric Methods Derived by Variational Means
- A modified BFGS method and its global convergence in nonconvex minimization
- A new approach to variable metric algorithms
- A reliable effective terascale linear learning system
- A robust multi-batch L-BFGS method for machine learning
- A Stochastic Approximation Method
- A stochastic quasi-Newton method for large-scale optimization
- Adaptive sampling strategies for stochastic optimization
- Algorithms for nonlinear constraints that use lagrangian functions
- An investigation of Newton-sketch and subsampled Newton methods
- Conditioning of Quasi-Newton Methods for Function Minimization
- Convergence Properties of the BFGS Algoritm
- Convergence rate of incremental subgradient algorithms
- Deep learning
- Distributed asynchronous deterministic and stochastic gradient optimization algorithms
- Exact and inexact subsampled Newton methods for optimization
- Global convergence of online limited memory BFGS
- scientific article; zbMATH DE number 51132 (Why is no real title available?)
- scientific article; zbMATH DE number 3529352 (Why is no real title available?)
- Hybrid deterministic-stochastic methods for data fitting
- Numerical Optimization
- On the global convergence of the BFGS method for nonconvex unconstrained optimization problems
- On the limited memory BFGS method for large scale optimization
- On the use of stochastic Hessian information in optimization methods for machine learning
- Optimization methods for large-scale machine learning
- Perturbed iterate analysis for asynchronous stochastic optimization
- Quasi-Newton Methods and their Application to Function Minimisation
- Quasi-Newton methods for machine learning: forget the past, just sample
- Redundancy techniques for straggler mitigation in distributed optimization and learning
- Sample size selection in optimization methods for machine learning
- SGD-QN: careful quasi-Newton stochastic gradient descent
- Single-machine and parallel-machine serial-batching scheduling problems with position-based learning effect and linear setup time
- Stochastic First- and Zeroth-Order Methods for Nonconvex Stochastic Programming
- Stochastic Quasi-Newton Methods for Nonconvex Stochastic Optimization
- The BFGS method with exact line searches fails for non-convex objective functions
- Updating Quasi-Newton Matrices with Limited Storage
Cited in
(16)- Diagonally scaled memoryless quasi-Newton methods with application to compressed sensing
- Limited-memory BFGS with displacement aggregation
- Nonmonotone diagonally scaled limited-memory BFGS methods with application to compressive sensing based on a penalty model
- A robust multi-batch L-BFGS method for machine learning
- Quasi-Newton methods for machine learning: forget the past, just sample
- Trust-region algorithms for training responses: machine learning methods using indefinite Hessian approximations
- LSOS: Line-search second-order stochastic optimization methods for nonconvex finite sums
- A machine-learning-accelerated distributed LBFGS method for field development optimization: algorithm, validation, and applications
- An overview of stochastic quasi-Newton methods for large-scale machine learning
- Globally Convergent Multilevel Training of Deep Residual Networks
- Adaptive sampling quasi-Newton methods for zeroth-order stochastic optimization
- A non-monotone trust-region method with noisy oracles and additional sampling
- Tuning parameters of deep neural network training algorithms pays off: a computational study
- Fully stochastic trust-region methods with Barzilai-Borwein steplengths
- A structured L-BFGS method with diagonal scaling and its application to image registration
- Diagonally scaled memoryless symmetric rank-one methods with application to compressed sensing
This page was built for publication: A robust multi-batch L-BFGS method for machine learning
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q4972551)
