Adaptive sampling strategies for stochastic optimization
From MaRDI portal
Abstract: In this paper, we propose a stochastic optimization method that adaptively controls the sample size used in the computation of gradient approximations. Unlike other variance reduction techniques that either require additional storage or the regular computation of full gradients, the proposed method reduces variance by increasing the sample size as needed. The decision to increase the sample size is governed by an inner product test that ensures that search directions are descent directions with high probability. We show that the inner product test improves upon the well known norm test, and can be used as a basis for an algorithm that is globally convergent on nonconvex functions and enjoys a global linear rate of convergence on strongly convex functions. Numerical experiments on logistic regression problems illustrate the performance of the algorithm.
Recommendations
- Adaptive sampling for incremental optimization using stochastic gradient descent
- On the adaptivity of stochastic gradient-based optimization
- Adaptive sampling quasi-Newton methods for zeroth-order stochastic optimization
- Sample size selection in optimization methods for machine learning
- Variable-sample methods for stochastic optimization
Cites work
- A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems
- Convex optimization algorithms
- Exploration-exploitation tradeoff using variance estimates in multi-armed bandits
- Global convergence rate analysis of unconstrained optimization methods based on probabilistic models
- scientific article; zbMATH DE number 2121575 (Why is no real title available?)
- Hybrid deterministic-stochastic methods for data fitting
- Introductory lectures on convex optimization. A basic course.
- Minimizing finite sums with the stochastic average gradient
- On sampling rates in simulation-based recursions
- On the Global Convergence of Trust Region Algorithms Using Inexact Gradient Information
- Optimal budget allocation for sample average approximation
- Optimization methods for large-scale machine learning
- Sample size selection in optimization methods for machine learning
- Stochastic First- and Zeroth-Order Methods for Nonconvex Stochastic Programming
- Variable-sample methods for stochastic optimization
Cited in
(66)- Alternative sampling strategy for a random optimization algorithm
- Online surrogate multiobjective optimization algorithm for contaminated groundwater remediation designs
- Resolving learning rates adaptively by locating stochastic non-negative associated gradient projection points using line searches
- Adaptive sampling line search for local stochastic optimization with integer variables
- Ritz-like values in steplength selections for stochastic gradient methods
- Iteratively sampling scheme for stochastic optimization with variable number sample path
- Parameter calibration in wake effect simulation model with stochastic gradient descent and stratified sampling
- Subsampled nonmonotone spectral gradient methods
- Improved sampling strategies for ensemble-based optimization
- Statistically equivalent surrogate material models: impact of random imperfections on the elasto-plastic response
- Risk-averse design of tall buildings for uncertain wind conditions
- Discriminative Bayesian filtering lends momentum to the stochastic Newton method for minimizing log-convex functions
- Adaptive sampling for incremental optimization using stochastic gradient descent
- Boosted sampling
- Randomized approach to nonlinear inversion combining random and optimized simultaneous sources and detectors
- A robust multi-batch L-BFGS method for machine learning
- Adaptive gradient-free method for stochastic optimization
- A Variable Sample-Size Stochastic Quasi-Newton Method for Smooth and Nonsmooth Stochastic Convex Optimization
- Large Sample Mean-Field Stochastic Optimization
- On the adaptivity of stochastic gradient-based optimization
- A stochastic line search method with expected complexity analysis
- Adaptive sequential sample average approximation for solving two-stage stochastic linear programs
- An Adaptive Sampling and Domain Learning Strategy for Multivariate Function Approximation on Unknown Domains
- An adaptive stochastic sequential quadratic programming with differentiable exact augmented Lagrangians
- A stochastic gradient method for a class of nonlinear PDE-constrained optimal control problems under uncertainty
- Inequality constrained stochastic nonlinear optimization via active-set sequential quadratic programming
- A trust region method for noisy unconstrained optimization
- Combining resampling and reweighting for faithful stochastic optimization
- An adaptive sampling augmented Lagrangian method for stochastic optimization with deterministic constraints
- Gradient-based optimisation of the conditional-value-at-risk using the multi-level Monte Carlo method
- An overview of stochastic quasi-Newton methods for large-scale machine learning
- Globally Convergent Multilevel Training of Deep Residual Networks
- Adaptive sampling stochastic multigradient algorithm for stochastic multiobjective optimization
- A limited-memory BFGS-based differential evolution algorithm for optimal control of nonlinear systems with mixed control variables and probability constraints
- A line search based proximal stochastic gradient algorithm with dynamical variance reduction
- Failure-Informed Adaptive Sampling for PINNs
- Adaptive sampling quasi-Newton methods for zeroth-order stochastic optimization
- Adaptive sampling strategies for risk-averse stochastic optimization with constraints
- A single timescale stochastic quasi-Newton method for stochastic optimization
- A stochastic gradient method with variance control and variable learning rate for deep learning
- Finite elements for Matérn-type random fields: uncertainty in computational mechanics and design optimization
- Estimating absorption and scattering in quantitative photoacoustic tomography with an adaptive Monte Carlo method for light transport
- Constrained and composite optimization via adaptive sampling methods
- Bolstering stochastic gradient descent with model building
- An investigation of stochastic trust-region based algorithms for finite-sum minimization
- A proximal stochastic quasi-Newton algorithm with dynamical sampling and stochastic line search
- An adaptive importance sampling algorithm for risk-averse optimization
- Variable metric proximal stochastic gradient methods with additional sampling
- A line-search based SGD algorithm with adaptive importance sampling
- Distributed stochastic constrained optimization with constant step-sizes via saddle-point dynamics
- Line-search based optimization using function approximations with tunable accuracy
- Convergence analysis of a proximal stochastic gradient algorithm with adaptive sampling for non-convex and non-smooth composite optimization problems
- Convergence of ease-controlled random reshuffling gradient algorithms under Lipschitz smoothness
- Spectral stochastic gradient method with additional sampling for finite and infinite sums
- Convergence analysis for a nonlocal gradient descent method via directional Gaussian smoothing
- Adaptive regularized quasi-Newton method using inexact first-order information
- A variable metric proximal stochastic gradient method: an application to classification problems
- Sample size selection in optimization methods for machine learning
- SLiSeS: subsampled line search spectral gradient method for finite sums
- Exploiting negative curvature in conjunction with adaptive sampling: theoretical results and a practical algorithm
- Adaptive sampling and step size for momentum proximal stochastic gradient method in nonconvex nonsmooth problems
- A survey of trust-region radius update mechanisms. Part I: First-order analysis
- Multi-iteration stochastic optimizers
- Adaptive-batch stochastic gradient descent for constrained optimization based on relaxed barrier functions
- A proximal Newton-type algorithm for zeroth-order stochastic composite optimization with a new norm test for sample size selection
- A stochastic conjugate gradient algorithm with Armijo technique for machine learning
This page was built for publication: Adaptive sampling strategies for stochastic optimization
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q4562248)
