Probabilistic line searches for stochastic optimization
From MaRDI portal
Abstract: In deterministic optimization, line searches are a standard tool ensuring stability and efficiency. Where only stochastic gradients are available, no direct equivalent has so far been formulated, because uncertain gradients do not allow for a strict sequence of decisions collapsing the search space. We construct a probabilistic line search by combining the structure of existing deterministic methods with notions from Bayesian optimization. Our method retains a Gaussian process surrogate of the univariate optimization objective, and uses a probabilistic belief over the Wolfe conditions to monitor the descent. The algorithm has very low computational cost, and no user-controlled parameters. Experiments show that it effectively removes the need to define a learning rate for stochastic gradient descent.
Recommendations
Cites work
- A Family of Variable-Metric Methods Derived by Variational Means
- A new approach to variable metric algorithms
- A New Method of Solving Nonlinear Simultaneous Equations
- A Stochastic Approximation Method
- Adaptive stepsizes for recursive estimation with applications in approximate dynamic programming
- Adaptive subgradient methods for online learning and stochastic optimization
- Bayesian filtering and smoothing
- Conditioning of Quasi-Newton Methods for Function Minimization
- Convergence Conditions for Ascent Methods
- Deep learning
- Efficient global optimization of expensive black-box functions
- Function minimization by conjugate gradients
- Gaussian processes for machine learning.
- scientific article; zbMATH DE number 6377992 (Why is no real title available?)
- scientific article; zbMATH DE number 4041188 (Why is no real title available?)
- scientific article; zbMATH DE number 3751955 (Why is no real title available?)
- scientific article; zbMATH DE number 45848 (Why is no real title available?)
- scientific article; zbMATH DE number 3304505 (Why is no real title available?)
- Information-Theoretic Regret Bounds for Gaussian Process Optimization in the Bandit Setting
- Large-scale machine learning with stochastic gradient descent
- Minimization of functions having Lipschitz continuous first partial derivatives
- Numerical Optimization
- Probabilistic line searches for stochastic optimization
Cited in
(21)- Resolving learning rates adaptively by locating stochastic non-negative associated gradient projection points using line searches
- Data-driven algorithm selection and tuning in optimization and signal processing
- Accelerating mini-batch SARAH by step size rules
- An empirical study into finding optima in stochastic optimization of neural networks
- Backtracking gradient descent method and some applications in large scale optimisation. II: Algorithms and experiments
- A modern retrospective on probabilistic numerics
- Stochastic quasi-Newton with line-search regularisation
- Discriminative Bayesian filtering lends momentum to the stochastic Newton method for minimizing log-convex functions
- Variance-based extragradient methods with line search for stochastic variational inequalities
- A unifying framework for Gaussian process pseudo-point approximations using power expectation propagation
- Probabilistic line searches for stochastic optimization
- A stochastic line search method with expected complexity analysis
- Improved variance reduction extragradient method with line search for stochastic variational inequalities
- An overview of stochastic quasi-Newton methods for large-scale machine learning
- Adaptive step size rules for stochastic optimization in large-scale learning
- A fast non-monotone line search for stochastic gradient descent
- High probability complexity bounds for adaptive step search based on stochastic oracles
- Bolstering stochastic gradient descent with model building
- To raise or not to raise: the autonomous learning rate question
- A line-search based SGD algorithm with adaptive importance sampling
- Stabilizing optimal control for nonlinear stochastic systems: a parametric gradient-based approach
This page was built for publication: Probabilistic line searches for stochastic optimization
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q4637044)
