Optimization methods for large-scale machine learning
From MaRDI portal
algorithm complexity analysismachine learningnoise reduction methodsnumerical optimizationsecond-order methodsstochastic gradient methods
Optimization of shapes other than minimal surfaces (49Q10) Numerical mathematical programming methods (65K05) Analysis of algorithms and problem complexity (68Q25) Learning and adaptive systems in artificial intelligence (68T05) Large-scale problems in mathematical programming (90C06) Nonlinear programming (90C30)
Abstract: This paper provides a review and commentary on the past, present, and future of numerical optimization algorithms in the context of machine learning applications. Through case studies on text classification and the training of deep neural networks, we discuss how optimization problems arise in machine learning and what makes them challenging. A major theme of our study is that large-scale machine learning represents a distinctive setting in which the stochastic gradient (SG) method has traditionally played a central role while conventional gradient-based nonlinear optimization techniques typically falter. Based on this viewpoint, we present a comprehensive theory of a straightforward, yet versatile SG algorithm, discuss its practical behavior, and highlight opportunities for designing algorithms with improved performance. This leads to a discussion about the next generation of optimization methods for large-scale machine learning, including an investigation of two main streams of research on techniques that diminish noise in the stochastic directions and methods that make use of second-order derivative approximations.
Recommendations
- On the use of stochastic Hessian information in optimization methods for machine learning
- Stochastic optimization for large-scale machine learning
- Stochastic dual coordinate ascent methods for regularized loss minimization
- Large-scale machine learning with stochastic gradient descent
- scientific article; zbMATH DE number 1786133
Cites work
- A Characterization of Superlinear Convergence and Its Application to Quasi-Newton Methods
- A Convergent Incremental Gradient Method with a Constant Step Size
- A coordinate gradient descent method for nonsmooth separable minimization
- A dual algorithm for the solution of nonlinear variational problems via finite element approximation
- A family of second-order methods for convex \(\ell _1\)-regularized optimization
- A Family of Variable-Metric Methods Derived by Variational Means
- A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems
- A new approach to variable metric algorithms
- A simulation-based approach to two-stage stochastic programming with recourse
- A Stochastic Approximation Method
- Acceleration of Stochastic Approximation by Averaging
- Adaptive subgradient methods for online learning and stochastic optimization
- Algorithm 778: L-BFGS-B
- An Asynchronous Parallel Stochastic Coordinate Descent Algorithm
- An inexact successive quadratic approximation method for L-1 regularized optimization
- An iterative thresholding algorithm for linear inverse problems with a sparsity constraint
- Conditioning of Quasi-Newton Methods for Function Minimization
- Convex optimization algorithms
- De-noising by soft-thresholding
- Efficiency of coordinate descent methods on huge-scale optimization problems
- Erratum: SGDQN is less careful than expected
- Estimation of dependences based on empirical data. Transl. from the Russian by Samuel Kotz
- Exact matrix completion via convex optimization
- Fisher's Method of Scoring
- scientific article; zbMATH DE number 6377992 (Why is no real title available?)
- scientific article; zbMATH DE number 3643047 (Why is no real title available?)
- scientific article; zbMATH DE number 3850830 (Why is no real title available?)
- scientific article; zbMATH DE number 51132 (Why is no real title available?)
- scientific article; zbMATH DE number 3567782 (Why is no real title available?)
- scientific article; zbMATH DE number 3574917 (Why is no real title available?)
- scientific article; zbMATH DE number 1206370 (Why is no real title available?)
- scientific article; zbMATH DE number 1332320 (Why is no real title available?)
- scientific article; zbMATH DE number 1569102 (Why is no real title available?)
- scientific article; zbMATH DE number 1569104 (Why is no real title available?)
- scientific article; zbMATH DE number 3446442 (Why is no real title available?)
- scientific article; zbMATH DE number 3449561 (Why is no real title available?)
- scientific article; zbMATH DE number 1843019 (Why is no real title available?)
- scientific article; zbMATH DE number 845714 (Why is no real title available?)
- scientific article; zbMATH DE number 852532 (Why is no real title available?)
- scientific article; zbMATH DE number 928863 (Why is no real title available?)
- scientific article; zbMATH DE number 6159604 (Why is no real title available?)
- scientific article; zbMATH DE number 7306852 (Why is no real title available?)
- scientific article; zbMATH DE number 5060482 (Why is no real title available?)
- scientific article; zbMATH DE number 3229153 (Why is no real title available?)
- scientific article; zbMATH DE number 3391741 (Why is no real title available?)
- Hybrid deterministic-stochastic methods for data fitting
- Incremental Least Squares Methods and the Extended Kalman Filter
- Inexact Newton Methods
- Information-Based Complexity, Feedback and Dynamics in Convex Programming
- Information-Theoretic Lower Bounds on the Oracle Complexity of Stochastic Convex Optimization
- Introductory lectures on convex optimization. A basic course.
- Iterative Solution of Nonlinear Equations in Several Variables
- Large-scale machine learning with stochastic gradient descent
- Learning representations by back-propagating errors
- LIBLINEAR: a library for large linear classification
- Minimizing finite sums with the stochastic average gradient
- Natural Langevin dynamics for neural networks
- New Classes of Synchronous Codes
- New method of stochastic approximation type
- Newton Sketch: A Near Linear-Time Optimization Algorithm with Linear-Quadratic Convergence
- Newton's Method for Large Bound-Constrained Optimization Problems
- On a Stochastic Approximation Method
- On perturbed proximal gradient algorithms
- On sampling rates in simulation-based recursions
- On search directions for minimization algorithms
- On the convergence properties of the EM algorithm
- On the Convergence Rate of Incremental Aggregated Gradient Algorithms
- On the Douglas-Rachford splitting method and the proximal point algorithm for maximal monotone operators
- On the Generalization Ability of On-Line Learning Algorithms
- On the limited memory BFGS method for large scale optimization
- On the use of stochastic Hessian information in optimization methods for machine learning
- On‐line learning for very large data sets
- Optimal aggregation of classifiers in statistical learning.
- Optimization for simulation: theory vs. practice
- Optimization with sparsity-inducing penalties
- Pegasos: primal estimated sub-gradient solver for SVM
- Practical inexact proximal quasi-Newton method with global complexity analysis
- Primal-dual subgradient methods for convex problems
- Probability and finance. It's only a game!
- Probability Inequalities for Sums of Bounded Random Variables
- Randomized methods for linear constraints: convergence rates and conditioning
- RES: Regularized Stochastic BFGS Algorithm
- Robust Stochastic Approximation Approach to Stochastic Programming
- Sample size selection in optimization methods for machine learning
- Second-order stochastic optimization for machine learning in linear time
- SGD-QN: careful quasi-Newton stochastic gradient descent
- Some applications of concentration inequalities to statistics
- Some methods of speeding up the convergence of iteration methods
- Sparse Reconstruction by Separable Approximation
- Stochastic dual coordinate ascent methods for regularized loss minimization
- Sub-sampled Newton methods
- Support-vector networks
- Templates for convex cone problems with applications to sparse signal recovery
- The Conjugate Gradient Method and Trust Regions in Large Scale Optimization
- The Convergence of a Class of Double-rank Minimization Algorithms 1. General Considerations
- The importance of convexity in learning with squared loss
- Trust Region Methods
- Uniform Central Limit Theorems
- Universal Portfolios
- Updating Quasi-Newton Matrices with Limited Storage
Cited in
(only showing first 100 items - show all)- A Levenberg-Marquardt method for large nonlinear least-squares problems with dynamic accuracy in functions and gradients
- On principal components regression, random projections, and column subsampling
- Stochastic optimization using a trust-region method and random models
- On variance reduction for stochastic smooth convex optimization with multiplicative noise
- Parallel decomposition methods for linearly constrained problems subject to simple bound with application to the SVMs training
- Stochastic gradient descent with Polyak's learning rate
- Forward stability of ResNet and its variants
- ADMM-softmax: an ADMM approach for multinomial logistic regression
- Stochastic variance reduced gradient methods using a trust-region-like scheme
- Selection dynamics for deep neural networks
- Convergence of stochastic proximal gradient algorithm
- PPINN: parareal physics-informed neural network for time-dependent PDEs
- Analysis of biased stochastic gradient descent using sequential semidefinite programs
- A variation of Broyden class methods using Householder adaptive transforms
- Convergence of stochastic gradient descent in deep neural network
- Convergence analysis of neural networks for solving a free boundary problem
- Stochastic proximal gradient methods for nonconvex problems in Hilbert spaces
- Convergence rates for optimised adaptive importance samplers
- Optimization problems for machine learning: a survey
- A unified convergence analysis of stochastic Bregman proximal gradient and extragradient methods
- General convergence analysis of stochastic first-order methods for composite optimization
- Non-convergence of stochastic gradient descent in the training of deep neural networks
- Conservative set valued fields, automatic differentiation, stochastic gradient methods and deep learning
- Computing Lyapunov functions using deep neural networks
- A stochastic subspace approach to gradient-free optimization in high dimensions
- Bias of homotopic gradient descent for the hinge loss
- Regularization parameter selection for the low rank matrix recovery
- Incremental without replacement sampling in nonconvex optimization
- Stochastic generalized gradient methods for training nonconvex nonsmooth neural networks
- Sequential convergence of AdaGrad algorithm for smooth convex optimization
- The generalized equivalence of regularization and min-max robustification in linear mixed models
- Adaptive optimization with periodic dither signals
- On large-scale unconstrained optimization and arbitrary regularization
- On the inexact scaled gradient projection method
- Finding best approximation pairs for two intersections of closed convex sets
- LSPIA, (stochastic) gradient descent, and parameter correction
- Quasi-convex feasibility problems: subgradient methods and convergence rates
- Remove the salt and pepper noise based on the high order total variation and the nuclear norm regularization
- Quantized convolutional neural networks through the lens of partial differential equations
- Convergence results of a nested decentralized gradient method for non-strongly convex problems
- On the local convergence of a stochastic semismooth Newton method for nonsmooth nonconvex optimization
- Dimension independent excess risk by stochastic gradient descent
- Variable metric proximal stochastic variance reduced gradient methods for nonconvex nonsmooth optimization
- SRKCD: a stabilized Runge-Kutta method for stochastic optimization
- Stopping criteria for, and strong convergence of, stochastic gradient descent on Bottou-Curtis-Nocedal functions
- Adaptive machine learning-based surrogate modeling to accelerate PDE-constrained optimization in enhanced oil recovery
- SABRINA: a stochastic subspace majorization-minimization algorithm
- Adaptive sampling line search for local stochastic optimization with integer variables
- Finite-sample analysis of nonlinear stochastic approximation with applications in reinforcement learning
- On the convergence of a block-coordinate incremental gradient method
- Triangularized orthogonalization-free method for solving extreme eigenvalue problems
- Tackling algorithmic bias in neural-network classifiers using Wasserstein-2 regularization
- A stochastic gradient algorithm with momentum terms for optimal control problems governed by a convection-diffusion equation with random diffusivity
- A subsampling approach for Bayesian model selection
- Convergence analysis of machine learning algorithms for the numerical solution of mean field control and games. II: The finite horizon case
- A stochastic first-order trust-region method with inexact restoration for finite-sum minimization
- A nested primal-dual FISTA-like scheme for composite convex optimization problems
- Inertial accelerated SGD algorithms for solving large-scale lower-rank tensor CP decomposition problems
- A deep domain decomposition method based on Fourier features
- Generating Nesterov's accelerated gradient algorithm by using optimal control theory for optimization
- Accelerating variance-reduced stochastic gradient methods
- Adaptive two-layer ReLU neural network. I: Best least-squares approximation
- Adaptive two-layer ReLU neural network. II: Ritz approximation to elliptic PDEs
- Stochastic gradient descent for semilinear elliptic equations with uncertainties
- Accelerating mini-batch SARAH by step size rules
- ODE-RU: a dynamical system view on recurrent neural networks
- Stochastic quasi-subgradient method for stochastic quasi-convex feasibility problems
- An online conjugate gradient algorithm for large-scale data analysis in machine learning
- On obtaining sparse semantic solutions for inverse problems, control, and neural network training
- Self-adaptive deep neural network: numerical approximation to functions and PDEs
- The mixed deep energy method for resolving concentration features in finite strain hyperelasticity
- Adaptive deep density approximation for Fokker-Planck equations
- Feasibility-based fixed point networks
- Online statistical inference for parameters estimation with linear-equality constraints
- A stochastic extra-step quasi-Newton method for nonsmooth nonconvex optimization
- Finite-sum smooth optimization with SARAH
- Ritz-like values in steplength selections for stochastic gradient methods
- Model order reduction method based on (r)POD-ANNs for parameterized time-dependent partial differential equations
- Retracted: Model order reduction method based on machine learning for parameterized time-dependent partial differential equations
- Sub-linear convergence of a stochastic proximal iteration method in Hilbert space
- Interpreting rate-distortion of variational autoencoder and using model uncertainty for anomaly detection
- A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functions
- Laplacian smoothing gradient descent
- Block layer decomposition schemes for training deep neural networks
- A distributed conjugate gradient online learning method over networks
- Subsampled nonmonotone spectral gradient methods
- A regularization interpretation of the proximal point method for weakly convex functions
- SHOPPER: a probabilistic model of consumer choice with substitutes and complements
- Accelerating incremental gradient optimization with curvature information
- Two-point step size gradient method for solving a deep learning problem
- Inexact restoration with subsampled trust-region methods for finite-sum minimization
- Recursive estimation for sparse Gaussian process regression
- A Bayesian perspective of statistical machine learning for big data
- Newton-type methods for non-convex optimization under inexact Hessian information
- Parallel sequential Monte Carlo for stochastic gradient-free nonconvex optimization
- ROCKET: exceptionally fast and accurate time series classification using random convolutional kernels
- A gradient descent method for solving a system of nonlinear equations
- Generalized gradients in dynamic optimization, optimal control, and machine learning problems
- Optimization for deep learning: an overview
- A review on deep learning in medical image reconstruction
Describes a project that uses
Uses Software
This page was built for publication: Optimization methods for large-scale machine learning
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q4641709)
