Homotopy relaxation training algorithms for infinite-width two-layer ReLU neural networks

DOI10.1007/S10915-024-02761-5MaRDI QIDQ6665313zbMATH OpenFDO

Authors Yahong Yang, Qipin Chen, Wenrui Hao

Publication date 17 January 2025

Published in Journal of Scientific Computing (Search for Journal in Brave)

Full work available at URL https://doi.org/10.1007/S10915-024-02761-5

zbMATH Keywords

homotopyneural networksoptimizationrelaxation

Mathematics Subject Classification ID

Numerical methods for mathematical programming, optimization and variational techniques (65K99) Artificial neural networks and deep learning (68T07) Parallel algorithms in computer science (68W10)

Recommendations

• Improving the expressive power of deep neural networks through integral activation transform
• Infinite-width limit of deep linear neural networks
• DANTE: deep alternations for training neural networks
• Magnitude and angle dynamics in training single ReLU neurons

Cites work

• A homotopy method for parameter estimation of nonlinear differential equations with multiple optima
• A homotopy training algorithm for fully connected neural networks
• Accelerated optimization with orthogonality constraints
• Adaptive activation functions accelerate convergence in deep and physics-informed neural networks
• An adaptive homotopy tracking algorithm for solving nonlinear parametric systems with applications in nonlinear ODEs
• Computing all solutions to polynomial systems using homotopy continuation
• Greedy training algorithms for neural networks and applications to PDEs
• High-dimensional probability. An introduction with applications in data science
• Locally adaptive activation functions with slope recovery for deep and physics-informed neural networks
• Neural tangent kernel: convergence and generalization in neural networks (invited paper)
• Phase diagram for two-layer ReLU neural networks at infinite-width limit
• Scaling Limit of the Stein Variational Gradient Descent: The Mean Field Regime
• Side effects of learning from low-dimensional data embedded in a Euclidean space
• Sobolev training of thermodynamic-informed neural networks for interpretable elasto-plasticity models with level set hardening
• The deep Ritz method: a deep learning-based numerical algorithm for solving variational problems
• The Numerical Solution of Systems of Polynomials Arising in Engineering and Science
• Why does unsupervised pre-training help deep learning?

Cited in

(4)

• Advancing neural network calibration: the role of gradient decay in large-margin Softmax optimization
• Laplacian eigenfunction-based neural operator for learning nonlinear reaction-diffusion dynamics
• Automatic differentiation is essential in training neural networks for solving differential equations
• Multiscale neural networks for approximating Green's functions

This page was built for publication: Homotopy relaxation training algorithms for infinite-width two-layer ReLU neural networks

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6665313)