Robust implicit regularization via weight normalization

DOI10.1093/IMAIAI/IAAE022MaRDI QIDQ6638528zbMATH OpenFDO

Authors Hung-Hsu Chou, Holger Rauhut, Rachel Ward

Publication date 14 November 2024

Published in Information and Inference: A Journal of the IMA (Search for Journal in Brave)

Copyright license Creative Commons Attribution 4.0 International

Full work available at URL https://doi.org/10.1093/IMAIAI/IAAE022

zbMATH Keywords

compressed sensinggradient descentimplicit regularizationL1 minimizationlinear neural networkoverparameterizationvector factorizationweight normalization

Mathematics Subject Classification ID

Numerical optimization and variational techniques (65K10) Artificial neural networks and deep learning (68T07) Applications of mathematical programming (90C90)

Recommendations

• The implicit bias of gradient descent on separable data
• Gradient descent for deep matrix factorization: dynamics and implicit bias towards low rank
• More is Less: Inducing Sparsity via Overparameterization
• Theoretical issues in deep networks
• Overparameterization and generalization error: weighted trigonometric interpolation

Cited in

(3)

• Stochastic modified flows for Riemannian stochastic gradient descent
• Convergence of gradient descent for recurrent neural networks: a nonasymptotic analysis
• Optimization over a probability simplex

This page was built for publication: Robust implicit regularization via weight normalization

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6638528)