Efficient and stable SAV-based methods for gradient flows arising from deep learning

DOI10.1016/J.JCP.2024.112911MaRDI QIDQ6497260zbMATH OpenFDO

Authors Ziqi Ma, Zhiping Mao, Jie Shen

Publication date 6 May 2024

Published in Journal of Computational Physics (Search for Journal in Brave)

Full work available at URL https://doi.org/10.1016/J.JCP.2024.112911

zbMATH Keywords

Adam methodadaptive learning ratedeep learningenergy stablegradient flowSAV method

Mathematics Subject Classification ID

Parabolic equations and parabolic systems (35Kxx) Numerical methods for partial differential equations, initial value and time-dependent initial-boundary value problems (65Mxx) Artificial intelligence (68Txx)

Recommendations

• An adaptive gradient method with energy and momentum
• An Efficient and Robust Scalar Auxialiary Variable Based Algorithm for Discrete Gradient Systems Arising from Optimizations
• Deep learning as optimal control problems: models and numerical methods
• Deep relaxation: partial differential equations for optimizing deep neural networks
• Maximum principle based algorithms for deep learning

Cites work

• A mean field view of the landscape of two-layer neural networks
• A mean-field optimal control formulation of deep learning
• A proposal on machine learning via dynamical systems
• A Stochastic Approximation Method
• Adaptive subgradient methods for online learning and stochastic optimization
• Approximation by superpositions of a sigmoidal function
• Deep backward schemes for high-dimensional nonlinear PDEs
• DeepXDE: a deep learning library for solving differential equations
• DGM: a deep learning algorithm for solving partial differential equations
• Improving the accuracy and consistency of the scalar auxiliary variable (SAV) method with relaxation
• Learning representations by back-propagating errors
• Machine learning from a continuous viewpoint. I
• Maximum principle based algorithms for deep learning
• Mean field analysis of neural networks: a central limit theorem
• Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
• Solving high-dimensional partial differential equations using deep learning
• Stable architectures for deep neural networks
• The Barron space and the flow-induced function spaces for neural network models
• The scalar auxiliary variable (SAV) approach for gradient flows
• Universal approximation bounds for superpositions of a sigmoidal function

Cited in

(3)

• SAV-based entropy-dissipative schemes for a class of kinetic equations
• Stability analysis and error estimation based on difference spectral approximation for Allen-Cahn equation in a circular domain
• A machine learning method for solving PDEs with a robust initial learning rate based on Lagrange multiplier optimization

This page was built for publication: Efficient and stable SAV-based methods for gradient flows arising from deep learning

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6497260)