Theoretical analysis of Adam using hyperparameters close to one without Lipschitz smoothness

DOI10.1007/s11075-023-01575-0arXiv2206.13290MaRDI QIDQ6145578

Hideaki Iiduka

Publication date: 9 January 2024

Published in: Numerical Algorithms (Search for Journal in Brave)

Full work available at URL: https://arxiv.org/abs/2206.13290

zbMATH Keywords

nonconvex optimizationlearning ratebatch sizehyperparametersadaptive moment estimation

Mathematics Subject Classification ID

Numerical mathematical programming methods (65K05) Nonconvex programming, global optimization (90C26) Stochastic programming (90C15) Mesh generation, refinement, and adaptive methods for ordinary differential equations (65L50)

Cites Work

• Unnamed Item
• Unnamed Item
• Convex analysis and nonlinear optimization. Theory and examples
• Robust Stochastic Approximation Approach to Stochastic Programming
• Matrix Analysis
• Optimization Methods for Large-Scale Machine Learning
• Optimal Stochastic Approximation Algorithms for Strongly Convex Stochastic Composite Optimization I: A Generic Algorithmic Framework
• Optimal Stochastic Approximation Algorithms for Strongly Convex Stochastic Composite Optimization, II: Shrinking Procedures and Optimal Algorithms
• Some methods of speeding up the convergence of iteration methods
• A Stochastic Approximation Method

This page was built for publication: Theoretical analysis of Adam using hyperparameters close to one without Lipschitz smoothness