A multivariate adaptive gradient algorithm with reduced tuning efforts

DOI10.1016/J.NEUNET.2022.05.016MaRDI QIDQ6488713zbMATH OpenFDO

Authors Samer Saab, Khaled Saab, Shashi Phoha, Minghui Zhu, Asok Ray

Publication date 17 October 2023

Published in Neural Networks (Search for Journal in Brave)

Full work available at URL https://doi.org/10.1016/J.NEUNET.2022.05.016

zbMATH Keywords

adaptive learning ratedeep learninggradient descent optimization

Mathematics Subject Classification ID

Artificial neural networks and deep learning (68T07) Mathematical programming (90Cxx)

Recommendations

• An adaptive gradient method with energy and momentum
• On the adaptivity of stochastic gradient-based optimization
• scientific article; zbMATH DE number 7306906
• An adaptive Polyak heavy-ball method
• A stochastic gradient method with variance control and variable learning rate for deep learning

Cites work

• A literature survey of benchmark functions for global optimisation problems
• A New Class of Incremental Gradient Methods for Least Squares Problems
• A Stochastic Approximation Method
• Adaptive subgradient methods for online learning and stochastic optimization
• Analysis and design of optimization algorithms via integral quadratic constraints
• Batched Stochastic Gradient Descent with Weighted Sampling
• Convergence of the RMSProp deep learning method with penalty for nonconvex optimization
• Introductory lectures on convex optimization. A basic course.
• Smoothing neural network for L₀ regularized optimization problem with general convex constraints
• Stochastic First- and Zeroth-Order Methods for Nonconvex Stochastic Programming
• Stochastic gradient descent, weighted sampling, and the randomized Kaczmarz algorithm
• Tunability: importance of hyperparameters of machine learning algorithms

This page was built for publication: A multivariate adaptive gradient algorithm with reduced tuning efforts

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6488713)