The importance of better models in stochastic optimization

DOI10.1073/PNAS.1908018116MaRDI QIDQ5218482zbMATH OpenOpenAlexWikidataFDO

Authors Hilal Asi, John C. Duchi

Publication date 4 March 2020

Published in Proceedings of the National Academy of Sciences (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1903.08619

Mathematics Subject Classification ID

Nonparametric robustness (62G35) Stochastic programming (90C15)

Abstract: Standard stochastic optimization methods are brittle, sensitive to stepsize choices and other algorithmic parameters, and they exhibit instability outside of well-behaved families of objectives. To address these challenges, we investigate models for stochastic minimization and learning problems that exhibit better robustness to problem families and algorithmic parameters. With appropriately accurate models---which we call the aProx family---stochastic methods can be made stable, provably convergent and asymptotically optimal; even modeling that the objective is nonnegative is sufficient for this stability. We extend these results beyond convexity to weakly convex objectives, which include compositions of convex losses with smooth functions common in modern machine learning applications. We highlight the importance of robustness and accurate modeling with a careful experimental evaluation of convergence time and algorithm sensitivity.

Recommendations

• Stochastic model-based minimization of weakly convex functions
• Stability analysis for stochastic optimization problems
• SRKCD: a stabilized Runge-Kutta method for stochastic optimization
• Asymptotic optimality in stochastic optimization
• On the adaptivity of stochastic gradient-based optimization

Cited in

(11)

• Hybrid SGD algorithms to solve stochastic composite optimization problems with application in sparse portfolio selection problems
• A dual-based stochastic inexact algorithm for a class of stochastic nonsmooth convex composite problems
• scientific article; zbMATH DE number 7370566 (Why is no real title available?)
• Global convergence of model function based Bregman proximal minimization algorithms
• Bolstering stochastic gradient descent with model building
• A semismooth Newton stochastic proximal point algorithm with variance reduction
• Efficient algorithms for implementing incremental proximal-point methods
• Stochastic optimization over proximally smooth sets
• Stochastic variance-reduced prox-linear algorithms for nonconvex composite optimization
• SRKCD: a stabilized Runge-Kutta method for stochastic optimization
• A unified framework for stochastic optimization

This page was built for publication: The importance of better models in stochastic optimization

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5218482)