Gradient descent provably escapes saddle points in the training of shallow ReLU networks

DOI10.1007/S10957-024-02513-3MaRDI QIDQ6655804zbMATH OpenFDO

Authors Patrick Cheridito, Arnulf Jentzen, Florian Rossmannek

Publication date 27 December 2024

Published in Journal of Optimization Theory and Applications (Search for Journal in Brave)

Copyright license Creative Commons Attribution 4.0 International

zbMATH Keywords

nonconvex optimizationneural networksgradient descentcenter-stable manifolds

Mathematics Subject Classification ID

Numerical optimization and variational techniques (65K10) Artificial neural networks and deep learning (68T07) Nonconvex programming, global optimization (90C26)

Recommendations

• Gradient descent optimizes over-parameterized deep ReLU networks
• The global optimization geometry of shallow linear neural networks
• Landscape analysis for shallow neural networks: complete classification of critical points for affine target functions
• Non-differentiable saddle points and sub-optimal local minima exist for deep ReLU networks

Cites work

• scientific article; zbMATH DE number 3980052 (Why is no real title available?)
• A comparative analysis of optimization and generalization properties of two-layer neural network and random feature models under gradient descent dynamics
• A geometric analysis of phase retrieval
• A proof of convergence for gradient descent in the training of artificial neural networks for constant target functions
• Behavior of accelerated gradient methods near critical points of nonconvex functions
• Complete Dictionary Recovery Over the Sphere I: Overview and the Geometric Picture
• Convergence of the Iterates of Descent Methods for Analytic Cost Functions
• First-order methods almost always avoid strict saddle points
• Gradient descent only converges to minimizers: non-isolated critical points and invariant regions
• Gradient descent optimizes over-parameterized deep ReLU networks
• Landscape analysis for shallow neural networks: complete classification of critical points for affine target functions
• Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizers
• Measure theory and fine properties of functions
• Nonconvergence to unstable points in urn models and stochastic approximations
• Nonconvex Robust Low-Rank Matrix Recovery
• Splitting methods with variable metric for Kurdyka-Łojasiewicz functions and general convergence rates
• Spurious valleys in one-hidden-layer neural network optimization landscapes
• Stochastic subgradient method converges on tame functions
• The nonsmooth landscape of phase retrieval
• Theoretical Insights Into the Optimization Landscape of Over-Parameterized Shallow Neural Networks

Cited in

(4)

• A deterministic gradient-based approach to avoid saddle points
• Universality of gradient descent neural network training
• Kurdyka-Łojasiewicz exponent via Hadamard parametrization
• Limitations of neural network training due to numerical instability of backpropagation

This page was built for publication: Gradient descent provably escapes saddle points in the training of shallow ReLU networks

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6655804)