Understanding and Mitigating Gradient Flow Pathologies in Physics-Informed Neural Networks

DOI10.1137/20M1318043MaRDI QIDQ4958918

Yujun Teng, Sifan Wang, Paris Perdikaris

Publication date: 15 September 2021

Published in: SIAM Journal on Scientific Computing (Search for Journal in Brave)

Full work available at URL: https://arxiv.org/abs/2001.04536

zbMATH Keywords

optimizationdifferential equationscomputational physicsdeep learningstiff dynamics

Mathematics Subject Classification ID

Artificial intelligence (68T99) Numerical methods for partial differential equations, initial value and time-dependent initial-boundary value problems (65M99)

Related Items

Hybrid FEM-NN models: combining artificial neural networks with the finite element method, Enforcing exact physics in scientific machine learning: a data-driven exterior calculus on graphs, Simple computational strategies for more effective physics-informed neural networks modeling of turbulent natural convection, When and why PINNs fail to train: a neural tangent kernel perspective, Physics-informed neural networks for gravity field modeling of the Earth and Moon, Meta-learning PINN loss functions, Physics-informed neural networks for rarefied-gas dynamics: Poiseuille flow in the BGK approximation, Accelerating phase-field predictions via recurrent neural networks learning the microstructure evolution in latent space, Physics-informed neural network simulation of multiphase poroelasticity using stress-split sequential training, Improved architectures and training algorithms for deep operator networks, Physics-informed neural networks for inverse problems in supersonic flows, Solving multiscale steady radiative transfer equation using neural networks with uniform stability, Scientific machine learning through physics-informed neural networks: where we are and what's next, RPINNs: rectified-physics informed neural networks for solving stationary partial differential equations, Data-driven rogue waves and parameters discovery in nearly integrable \(\mathcal{PT}\)-symmetric Gross-Pitaevskii equations via PINNs deep learning, A-PINN: auxiliary physics informed neural networks for forward and inverse problems of nonlinear integro-differential equations, Active Neuron Least Squares: A Training Method for Multivariate Rectified Neural Networks, Physics and equality constrained artificial neural networks: application to forward and inverse problems with multi-fidelity data fusion, Improved deep neural networks with domain decomposition in solving partial differential equations, A combination of large eddy simulation and physics-informed machine learning to predict pore-scale flow behaviours in fibrous porous media: a case study of transient flow passing through a surgical mask, A physics-informed neural network technique based on a modified loss function for computational 2D and 3D solid mechanics, A unified scalable framework for causal sweeping strategies for physics-informed neural networks (PINNs) and their temporal decompositions, Physics-informed neural networks for 2nd order ODEs with sharp gradients, One-dimensional ice shelf hardness inversion: clustering behavior and collocation resampling in physics-informed neural networks, A-WPINN algorithm for the data-driven vector-soliton solutions and parameter discovery of general coupled nonlinear equations, Kolmogorov n-width and Lagrangian physics-informed neural networks: a causality-conforming manifold for convection-dominated PDEs, Physics-Informed Neural Networks for Solving Dynamic Two-Phase Interface Problems, VC-PINN: variable coefficient physics-informed neural network for forward and inverse problems of PDEs with variable coefficient, Enhanced physics‐informed neural networks for hyperelasticity, Enforcing continuous symmetries in physics-informed neural network for solving forward and inverse problems of partial differential equations, A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks, BINN: a deep learning approach for computational mechanics problems based on boundary integral equations, Adaptive weighting of Bayesian physics informed neural networks for multitask and multiscale forward and inverse problems, Computing non-equilibrium trajectories by a deep learning approach, Fully probabilistic deep models for forward and inverse problems in parametric PDEs, Solving Elliptic Problems with Singular Sources Using Singularity Splitting Deep Ritz Method, Deep Ritz method with adaptive quadrature for linear elasticity, Uncertainty quantification in scientific machine learning: methods, metrics, and comparisons, Pre-training strategy for solving evolution equations based on physics-informed neural networks, Physics-informed neural networks with adaptive localized artificial viscosity, Investigating and Mitigating Failure Modes in Physics-Informed Neural Networks (PINNs), Physics-informed neural networks combined with polynomial interpolation to solve nonlinear partial differential equations, Error convergence and engineering-guided hyperparameter search of PINNs: towards optimized I-FENN performance, Long-time integration of parametric evolution equations with physics-informed DeepONets, Capturing the diffusive behavior of the multiscale linear transport equations by asymptotic-preserving convolutional deeponets, Variable linear transformation improved physics-informed neural networks to solve thin-layer flow problems, Adaptive task decomposition physics-informed neural networks, Respecting causality for training physics-informed neural networks, Isogeometric neural networks: a new deep learning approach for solving parameterized partial differential equations, Transfer learning based physics-informed neural networks for solving inverse problems in engineering structures under different loading scenarios, Efficient physics-informed neural networks using hash encoding, Loss-attentional physics-informed neural networks, Pre-training physics-informed neural network with mixed sampling and its application in high-dimensional systems, A dive into spectral inference networks: improved algorithms for self-supervised learning of continuous spectral representations, Physics-informed neural networks for analysis of 2D thin-walled structures, A novel temperature prediction method without using energy equation based on physics-informed neural network (PINN): a case study on plate-circular/square pin-fin heat sinks, Learning Specialized Activation Functions for Physics-Informed Neural Networks, Bi-Orthogonal fPINN: A Physics-Informed Neural Network Method for Solving Time-Dependent Stochastic Fractional PDEs, Variable separated physics-informed neural networks based on adaptive weighted loss functions for blood flow model, An artificial neural network approach for a class of time-fractional diffusion and diffusion-wave equations, PINN training using biobjective optimization: the trade-off between data loss and residual loss, Solving free-surface problems for non-shallow water using boundary and initial conditions-free physics-informed neural network (bif-PINN), Improved training of physics-informed neural networks for parabolic differential equations with sharply perturbed initial conditions, PINN-FORM: a new physics-informed neural network for reliability analysis with partial differential equation, A symmetry group based supervised learning method for solving partial differential equations, Exact Dirichlet boundary physics-informed neural network EPINN for solid mechanics, Physics informed and data-based augmented learning in structural health diagnosis, Finite basis physics-informed neural networks (FBPINNs): a scalable domain decomposition approach for solving differential equations, Adaptive Learning Rate Residual Network Based on Physics-Informed for Solving Partial Differential Equations, Inverse modeling of nonisothermal multiphase poromechanics using physics-informed neural networks, Failure-Informed Adaptive Sampling for PINNs, A Neural Network Approach for Homogenization of Multiscale Problems, On the spectral bias of coupled frequency predictor-corrector triangular DNN: the convergence analysis, Discontinuity computing using physics-informed neural networks, Feature-adjacent multi-fidelity physics-informed machine learning for partial differential equations, High Order Deep Domain Decomposition Method for Solving High Frequency Interface Problems, An Adaptive Physics-Informed Neural Network with Two-Stage Learning Strategy to Solve Partial Differential Equations, Physics-informed machine learning method with space-time Karhunen-Loève expansions for forward and inverse partial differential equations, Data-driven nonparametric identification of material behavior based on physics-informed neural network with full-field data, Data-driven Whitney forms for structure-preserving control volume analysis, Physical informed neural networks with soft and hard boundary constraints for solving advection-diffusion equations using Fourier expansions, Residual-based attention in physics-informed neural networks, An unsupervised latent/output physics-informed convolutional-LSTM network for solving partial differential equations using peridynamic differential operator, Physics-informed neural networks based on adaptive weighted loss functions for Hamilton-Jacobi equations, An overview on deep learning-based approximation methods for partial differential equations, Physics informed neural networks: a case study for gas transport problems, Data-driven vector soliton solutions of coupled nonlinear Schrödinger equation using a deep learning algorithm, Locally adaptive activation functions with slope recovery for deep and physics-informed neural networks, Bayesian differential programming for robust systems identification under uncertainty, Interpretable machine learning: fundamental principles and 10 grand challenges, CENN: conservative energy method based on neural networks with subdomains for solving variational problems involving heterogeneous and complex geometries, Physics-informed neural networks for shell structures, Physics-informed neural networks for gravity field modeling of small bodies, Prediction of the number of solitons for initial value of nonlinear Schrödinger equation based on the deep learning method, Uniform convergence guarantees for the deep Ritz method for nonlinear problems, Self-adaptive physics-informed neural networks, Neural eikonal solver: improving accuracy of physics-informed neural networks for solving eikonal equation in case of caustics

Uses Software

• TensorFlow
• DiffSharp
• ImageNet
• Adam
• GitHub
• Tensor2Tensor
• AlexNet
• DGM
• PhyGeoNet
• PINNsNTK
• GradNorm
• hp-VPINNs

Cites Work

• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Multirate linear multistep methods
• PhyGeoNet
• Machine learning in cardiovascular flows modeling: predicting arterial blood pressure from non-invasive 4D flow MRI data using physics-informed neural networks
• Deep UQ: learning deep neural network surrogate models for high dimensional uncertainty quantification
• DGM: a deep learning algorithm for solving partial differential equations
• Surrogate modeling for fluid flows based on physics-constrained deep learning without simulation data
• Physics-constrained deep learning for high-dimensional surrogate modeling and uncertainty quantification without labeled data
• Adversarial uncertainty quantification in physics-informed neural networks
• Modeling the dynamics of PDE systems with physics-constrained deep auto-regressive networks
• Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
• The 2D lid-driven cavity problem revisited
• Nonintrusive and Structure Preserving Multiscale Integration of Stiff ODEs, SDEs, and Hamiltonian Systems with Hidden Slow Dynamics via Flow Averaging
• A First Course in the Numerical Analysis of Differential Equations
• Deep Hidden Physics Models: Deep Learning of Nonlinear Partial Differential Equations
• Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations
• On the Convergence of Physics Informed Neural Networks for Linear Second-Order Elliptic and Parabolic Type PDEs
• Spectral Methods
• On the Construction and Comparison of Difference Schemes
• Acoustic and electromagnetic equations. Integral representations for harmonic problems