Gradient descent provably escapes saddle points in the training of shallow ReLU networks
From MaRDI portal
(Redirected from Gradient descent provably escapes saddle points in the training of shallow ReLU networks)
Gradient descent provably escapes saddle points in the training of shallow ReLU networks (scientific article; zbMATH DE number 7960846)
Gradient descent provably escapes saddle points in the training of shallow ReLU networks (scientific article; zbMATH DE number 7960846)
Recommendations
- Gradient descent optimizes over-parameterized deep ReLU networks
- The global optimization geometry of shallow linear neural networks
- Landscape analysis for shallow neural networks: complete classification of critical points for affine target functions
- Non-differentiable saddle points and sub-optimal local minima exist for deep ReLU networks
Cites work
- A comparative analysis of optimization and generalization properties of two-layer neural network and random feature models under gradient descent dynamics
- A geometric analysis of phase retrieval
- A proof of convergence for gradient descent in the training of artificial neural networks for constant target functions
- Behavior of accelerated gradient methods near critical points of nonconvex functions
- Complete Dictionary Recovery Over the Sphere I: Overview and the Geometric Picture
- Convergence of the Iterates of Descent Methods for Analytic Cost Functions
- First-order methods almost always avoid strict saddle points
- Gradient descent only converges to minimizers: non-isolated critical points and invariant regions
- Gradient descent optimizes over-parameterized deep ReLU networks
- scientific article; zbMATH DE number 3980052 (Why is no real title available?)
- Landscape analysis for shallow neural networks: complete classification of critical points for affine target functions
- Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizers
- Measure theory and fine properties of functions
- Nonconvergence to unstable points in urn models and stochastic approximations
- Nonconvex Robust Low-Rank Matrix Recovery
- Splitting methods with variable metric for Kurdyka-Łojasiewicz functions and general convergence rates
- Spurious valleys in one-hidden-layer neural network optimization landscapes
- Stochastic subgradient method converges on tame functions
- The nonsmooth landscape of phase retrieval
- Theoretical Insights Into the Optimization Landscape of Over-Parameterized Shallow Neural Networks
Cited in
(6)- When does gradient descent with logistic loss find interpolating two-layer networks?
- Universality of gradient descent neural network training
- Limitations of neural network training due to numerical instability of backpropagation
- A deterministic gradient-based approach to avoid saddle points
- Kurdyka-Łojasiewicz exponent via Hadamard parametrization
- Endogenous regime switching driven by scalar-irreducible learning dynamics
This page was built for publication: Gradient descent provably escapes saddle points in the training of shallow ReLU networks
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6655804)
