Critical point-finding methods reveal gradient-flat regions of deep network losses

DOI10.1162/NECO_A_01388MaRDI QIDQ5004367zbMATH OpenOpenAlexFDO

Authors Charles G. Frye, James Simon, Neha S. Wadia, Andrew Ligeralde, Michael R DeWeese, Kristofer E. Bouchard

Publication date 30 July 2021

Published in Neural Computation (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2003.10397

Mathematics Subject Classification ID

Artificial intelligence (68Txx)

Recommendations

• Shaping the learning landscape in neural networks around wide flat minima
• Landscape analysis for shallow neural networks: complete classification of critical points for affine target functions
• Optimization Landscape of Neural Networks
• The global optimization geometry of shallow linear neural networks
• Symmetry \& critical points for a model shallow neural network

Cites work

• Adaptive subgradient methods for online learning and stochastic optimization
• Analysis of Newton’s Method at Irregular Singularities
• scientific article; zbMATH DE number 3436540 (Why is no real title available?)
• scientific article; zbMATH DE number 2095946 (Why is no real title available?)
• scientific article; zbMATH DE number 2107836 (Why is no real title available?)
• scientific article; zbMATH DE number 5060482 (Why is no real title available?)
• Implementable tensor methods in unconstrained convex optimization
• MINRES-QLP: a Krylov subspace method for indefinite or singular symmetric systems
• Newton-Type Methods for Optimization and Variational Problems
• Numerically solving polynomial systems with Bertini
• On the convergence of Newton iterations to non-stationary points
• Residual and backward error bounds in minimum residual Krylov subspace methods
• Scikit-learn: machine learning in Python
• The Fundamental Theorem of Linear Algebra

Cited in

(3)

• Shaping the learning landscape in neural networks around wide flat minima
• Newton-MR: inexact Newton method with minimum residual sub-problem solver
• Visualizing high-dimensional loss landscapes with Hessian directions

Describes a project that uses

Uses Software

• Bertini
• Scikit
• MINRES-QLP
• MNIST
• MINRES
• PyTorch
• AdaGrad
• Adam

This page was built for publication: Critical point-finding methods reveal gradient-flat regions of deep network losses

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5004367)