Magnitude and angle dynamics in training single ReLU neurons

DOI10.1016/J.NEUNET.2024.106435MaRDI QIDQ6587015zbMATH OpenFDO

Authors Sangmin Lee, Byeongsu Sim, Jong Chul Ye

Publication date 13 August 2024

Published in Neural Networks (Search for Journal in Brave)

Full work available at URL https://doi.org/10.1016/J.NEUNET.2024.106435

zbMATH Keywords

gradient flowmagnitude and angle dynamicssingle ReLU neurons

Mathematics Subject Classification ID

Artificial neural networks and deep learning (68T07)

Recommendations

• The Computational Complexity of ReLU Network Training Parameterized by Data Dimensionality
• Non-convergence of stochastic gradient descent in the training of deep neural networks
• Improved weight initialization for deep and narrow feedforward neural network
• Training neural networks from an ergodic perspective
• On the Relation Between Loss Functions and T-Norms

Cites work

• How Small Is a Unit Ball?
• scientific article; zbMATH DE number 1149836 (Why is no real title available?)
• The implicit bias of gradient descent on separable data
• Two models of double descent for weak features

Cited in

(3)

• Improved weight initialization for deep and narrow feedforward neural network
• Training neural networks from an ergodic perspective
• Homotopy relaxation training algorithms for infinite-width two-layer ReLU neural networks

This page was built for publication: Magnitude and angle dynamics in training single ReLU neurons

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6587015)