Theoretical analysis of Adam using hyperparameters close to one without Lipschitz smoothness (Q6145578)

Convergence rate analyses of Adaptive Moment Estimation are widely used in nonconvex optimsation. This paper analyses this method and shows that the method performs well for small learning rates and hyperparameters close to 1. This has implications for nonconvex optimisation for deep learning applications.