A control theoretic framework for adaptive gradient optimizers

DOI10.1016/J.AUTOMATICA.2023.111466MaRDI QIDQ6152585zbMATH OpenFDO

Authors Kushal Chakrabarti, Nikhil Chopra

Publication date 13 February 2024

Published in Automatica (Search for Journal in Brave)

Full work available at URL https://doi.org/10.1016/J.AUTOMATICA.2023.111466

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05) Nonconvex programming, global optimization (90C26) Adaptive control/observation systems (93C40)

Recommendations

• AdaLo: adaptive learning rate optimizer with loss for classification
• Theoretical analysis of Adam using hyperparameters close to one without Lipschitz smoothness
• A modification of adaptive moment estimation (Adam) for machine learning
• Efficient learning rate adaptation based on hierarchical optimization approach
• Adaptive methods using element-wise pth power of stochastic gradient for nonconvex optimization in deep neural networks

Cites work

• scientific article; zbMATH DE number 3850830 (Why is no real title available?)
• Adaptive subgradient methods for online learning and stochastic optimization
• An LMI approach to constrained optimization with homogeneous forms
• Convergence and dynamical behavior of the ADAM algorithm for nonconvex stochastic optimization
• Iterative pre-conditioning for expediting the distributed gradient-descent method: the case of linear least-squares problem
• Nonlinear systems.
• Resource-Aware Discretization of Accelerated Optimization Flows: The Heavy-Ball Dynamics Case
• Robust control. Systems with uncertain physical parameters. In co-operation with A. Bartlett, D. Kaesbauer, W. Sienel, R. Steinhauser
• Transient Growth of Accelerated Optimization Algorithms

Cited in

(3)

• Efficient learning rate adaptation based on hierarchical optimization approach
• Adaptive learning rate optimization algorithms with dynamic bound based on Barzilai-Borwein method
• AdaLo: adaptive learning rate optimizer with loss for classification

This page was built for publication: A control theoretic framework for adaptive gradient optimizers

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6152585)