Decoupled Weight Decay Regularization

DOI10.48550/ARXIV.1711.05101MaRDI QIDQ71633arXivFDO

Authors Ilya Loshchilov, Frank Hutter

Publication date 14 November 2017

Published in arXiv

Full work available at URL https://doi.org/10.48550/ARXIV.1711.05101

arXiv classification

cs.LGcs.NEmath.OC

Abstract: L${}_2$ regularization and weight decay regularization are equivalent for standard stochastic gradient descent (when rescaled by the learning rate), but as we demonstrate this is emph{not} the case for adaptive gradient algorithms, such as Adam. While common implementations of these algorithms employ L${}_2$ regularization (often calling it "weight decay" in what may be misleading due to the inequivalence we expose), we propose a simple modification to recover the original formulation of weight decay regularization by emph{decoupling} the weight decay from the optimization steps taken w.r.t. the loss function. We provide empirical evidence that our proposed modification (i) decouples the optimal choice of weight decay factor from the setting of the learning rate for both standard SGD and Adam and (ii) substantially improves Adam's generalization performance, allowing it to compete with SGD with momentum on image classification datasets (on which it was previously typically outperformed by the latter). Our proposed decoupled weight decay has already been adopted by many researchers, and the community has implemented it in TensorFlow and PyTorch; the complete source code for our experiments is available at https://github.com/loshchil/AdamW-and-SGDW

Cited in

(14)

• torchopt
• Tensorion: A Tensor-Aware Generalization of the Muon Optimizer
• A Hybrid Physics-Informed Neural Network Framework for Subcritical and Supercritical Dynamics in Multi-Species Chemotaxis
• Physics-informed token transformer methodology for nonlinear balance laws. I. Schwarzschild--Burgers fluid flows
• A Neural-network-based multiscale Hybridizable Discontinuous Galerkin method for solving PDEs in porous media
• Stiefel Attention: When the Geometry of Transformer Projection Matrices Dominates Optimizer Choice---and When It Does Not
• Integral chemical reaction neural networks
• Spectral Deflation for Factorization-Free Matrix Filtering in Muon and Semidefinite Programming
• KATOsuper: Surrogate-accelerated neural topology optimization with sensitivity-consistent Fourier neural operators
• AURA: Angular Update Rate Adaptation for training complex-valued neural networks
• Safety of Latent Communication in Multi-Agent Systems
• Inferring constitutive forces from noisy Allen-Cahn movies with moment equations
• Inferring physical fields in coupled systems with unknown parameters from incomplete observations using physics-constrained attentive neural operators
• Learning a generalized Navier-Stokes model beyond the continuum regime

This page was built for publication: Decoupled Weight Decay Regularization

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q71633)