Global convergence in learning fully-connected ReLU networks via un-rectifying based on the augmented Lagrangian approach

DOI10.1007/S10915-024-02548-8MaRDI QIDQ6536825zbMATH OpenFDO

Authors Shih-Shuo Tung, M. Y. Chung, Jinn Ho, Wen-Liang Hwang

Publication date 14 May 2024

Published in Journal of Scientific Computing (Search for Journal in Brave)

Full work available at URL https://doi.org/10.1007/S10915-024-02548-8

zbMATH Keywords

machine learningoptimizationun-rectification

Mathematics Subject Classification ID

Combinatorial optimization (90C27)

Recommendations

• Gradient descent optimizes over-parameterized deep ReLU networks
• Convergence of deep convolutional neural networks
• An augmented Lagrangian method for training recurrent neural networks
• scientific article; zbMATH DE number 1552027
• Constructive deep ReLU neural network approximation

Cites work

• A globally convergent algorithm for nonconvex optimization based on block coordinate update
• A Globally Convergent Augmented Lagrangian Algorithm for Optimization with General Constraints and Simple Bounds
• A proposal on machine learning via dynamical systems
• A Stochastic Approximation Method
• Adaptive subgradient methods for online learning and stochastic optimization
• ADMM for multiaffine constrained optimization
• Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward-backward splitting, and regularized Gauss-Seidel methods
• Global convergence of ADMM in nonconvex nonsmooth optimization
• Learning representations by back-propagating errors
• On gradients of functions definable in o-minimal structures
• On semi- and subanalytic geometry
• On the Convergence of Alternating Direction Lagrangian Methods for Nonconvex Structured Optimization Problems
• Optimization methods for large-scale machine learning
• Un-Rectifying Non-Linear Networks for Signal Representation
• Understanding the convergence of the alternating direction method of multipliers: theoretical and computational perspectives

Cited in

(1)

• Non-asymptotic estimates for TUSLA algorithm for non-convex learning with applications to neural networks with ReLU activation function

This page was built for publication: Global convergence in learning fully-connected ReLU networks via un-rectifying based on the augmented Lagrangian approach

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6536825)