Theoretical analysis of Adam using hyperparameters close to one without Lipschitz smoothness
From MaRDI portal
Abstract: Convergence and convergence rate analyses of adaptive methods, such as Adaptive Moment Estimation (Adam) and its variants, have been widely studied for nonconvex optimization. The analyses are based on assumptions that the expected or empirical average loss function is Lipschitz smooth (i.e., its gradient is Lipschitz continuous) and the learning rates depend on the Lipschitz constant of the Lipschitz continuous gradient. Meanwhile, numerical evaluations of Adam and its variants have clarified that using small constant learning rates without depending on the Lipschitz constant and hyperparameters ( and ) close to one is advantageous for training deep neural networks. Since computing the Lipschitz constant is NP-hard, the Lipschitz smoothness condition would be unrealistic. This paper provides theoretical analyses of Adam without assuming the Lipschitz smoothness condition in order to bridge the gap between theory and practice. The main contribution is to show theoretical evidence that Adam using small learning rates and hyperparameters close to one performs well, whereas the previous theoretical results were all for hyperparameters close to zero. Our analysis also leads to the finding that Adam performs well with large batch sizes. Moreover, we show that Adam performs well when it uses diminishing learning rates and hyperparameters close to one.
Recommendations
- Convergence and dynamical behavior of the ADAM algorithm for nonconvex stochastic optimization
- scientific article; zbMATH DE number 7306906
- Conjugate-gradient-based Adam for nonconvex stochastic optimization and its application to deep learning
- A modification of adaptive moment estimation (Adam) for machine learning
Cites work
- A Stochastic Approximation Method
- Adaptive subgradient methods for online learning and stochastic optimization
- Convergence rates for the stochastic gradient descent method for non-convex objective functions
- Convex analysis and nonlinear optimization. Theory and examples
- Matrix Analysis
- Optimal Stochastic Approximation Algorithms for Strongly Convex Stochastic Composite Optimization I: A Generic Algorithmic Framework
- Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization. II: Shrinking procedures and optimal algorithms
- Optimization methods for large-scale machine learning
- Robust Stochastic Approximation Approach to Stochastic Programming
- Some methods of speeding up the convergence of iteration methods
Cited in
(2)
This page was built for publication: Theoretical analysis of Adam using hyperparameters close to one without Lipschitz smoothness
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6145578)
