An adaptive Polyak heavy-ball method

DOI10.1007/S10994-022-06215-7MaRDI QIDQ2102380OpenAlexWikidataFDO

Authors Yanyan Li

Publication date 28 November 2022

Published in Machine Learning (Search for Journal in Brave)

Full work available at URL https://doi.org/10.1007/s10994-022-06215-7

zbMATH Keywords

global convergencegradient descentPolyak heavy-ball

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05)

Recommendations

• Stochastic heavy ball
• On the convergence analysis of aggregated heavy-ball method
• A robust control approach to asymptotic optimality of the heavy ball method for optimization of quadratic functions
• Convergence rates of the heavy ball method for quasi-strongly convex optimization
• Convergence rates for the heavy-ball continuous dynamics for non-convex optimization, under Polyak-Łojasiewicz condition

Cites work

• scientific article; zbMATH DE number 7306906 (Why is no real title available?)
• A New Class of Incremental Gradient Methods for Least Squares Problems
• A Stochastic Approximation Method
• A literature survey of benchmark functions for global optimisation problems
• Accelerated gradient methods for nonconvex nonlinear and stochastic programming
• Adaptive subgradient methods for online learning and stochastic optimization
• An accelerated randomized proximal coordinate gradient method and its application to regularized empirical risk minimization
• Analysis and design of optimization algorithms via integral quadratic constraints
• Automated adaptation strategies for stream learning
• First-order methods in optimization
• Heavy-ball method in nonconvex optimization problems
• ImageNet
• Introductory lectures on convex optimization. A basic course.
• Momentum and stochastic momentum for stochastic gradient, Newton, proximal point and subspace descent methods
• Some methods of speeding up the convergence of iteration methods
• Stochastic First- and Zeroth-Order Methods for Nonconvex Stochastic Programming
• Stochastic gradient descent with Polyak's learning rate
• Stochastic heavy ball
• Tunability: importance of hyperparameters of machine learning algorithms
• iPiano: inertial proximal algorithm for nonconvex optimization
• iPiasco: inertial proximal algorithm for strongly convex optimization

Cited in

(4)

• On adaptive stochastic heavy ball momentum for solving linear systems
• A multivariate adaptive gradient algorithm with reduced tuning efforts
• Non-monotone Behavior of the Heavy Ball Method
• Stochastic momentum methods for non-convex learning without bounded assumptions

Describes a project that uses

Uses Software

• iPiano
• MNIST
• iPiasco
• CIFAR
• PyTorch
• ImageNet
• AdaGrad
• Adam
• ADADELTA

This page was built for publication: An adaptive Polyak heavy-ball method

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2102380)