On the eigenvector bias of Fourier feature networks: from regression to solving multi-scale PDEs with physics-informed neural networks
From MaRDI portal
(Redirected from Publication:2237440)
deep learningneural tangent kernelpartial differential equationsscientific machine learningspectral bias
General topics in linear spectral theory for PDEs (35P05) Fourier and Fourier-Stieltjes transforms and other transforms of Fourier type (42B10) Spectral, collocation and related methods for boundary value problems involving PDEs (65N35) Learning and adaptive systems in artificial intelligence (68T05)
Abstract: Physics-informed neural networks (PINNs) are demonstrating remarkable promise in integrating physical models with gappy and noisy observational data, but they still struggle in cases where the target functions to be approximated exhibit high-frequency or multi-scale features. In this work we investigate this limitation through the lens of Neural Tangent Kernel (NTK) theory and elucidate how PINNs are biased towards learning functions along the dominant eigen-directions of their limiting NTK. Using this observation, we construct novel architectures that employ spatio-temporal and multi-scale random Fourier features, and justify how such coordinate embedding layers can lead to robust and accurate PINN models. Numerical examples are presented for several challenging cases where conventional PINN models fail, including wave propagation and reaction-diffusion dynamics, illustrating how the proposed methods can be used to effectively tackle both forward and inverse problems involving partial differential equations with multi-scale behavior. All code an data accompanying this manuscript will be made publicly available at url{https://github.com/PredictiveIntelligenceLab/MultiscalePINNs}.
Recommendations
- Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
- Physics-informed graph neural Galerkin networks: a unified framework for solving PDE-governed forward and inverse problems
- When do extended physics-informed neural networks (XPINNs) improve generalization?
- On the convergence of physics informed neural networks for linear second-order elliptic and parabolic type PDEs
- Investigating and Mitigating Failure Modes in Physics-Informed Neural Networks (PINNs)
Cites work
- Adversarial uncertainty quantification in physics-informed neural networks
- Bayesian differential programming for robust systems identification under uncertainty
- Deep hidden physics models: deep learning of nonlinear partial differential equations
- Exponential time differencing for stiff systems
- Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations
- scientific article; zbMATH DE number 1235880 (Why is no real title available?)
- scientific article; zbMATH DE number 6982909 (Why is no real title available?)
- Machine learning in cardiovascular flows modeling: predicting arterial blood pressure from non-invasive 4D flow MRI data using physics-informed neural networks
- Nonlinear analysis of hydrodynamic instability in laminar flames—I. Derivation of basic equations
- On the convergence of physics informed neural networks for linear second-order elliptic and parabolic type PDEs
- On the Eigenspectrum of the Gram Matrix and the Generalization Error of Kernel-PCA
- Physics-constrained deep learning for high-dimensional surrogate modeling and uncertainty quantification without labeled data
- Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
- Simulator-free solution of high-dimensional stochastic elliptic partial differential equations using deep neural networks
- Solving high-dimensional partial differential equations using deep learning
- Spectral Methods for Time-Dependent Problems
- Surrogate modeling for fluid flows based on physics-constrained deep learning without simulation data
- Wide neural networks of any depth evolve as linear models under gradient descent *
Cited in
(only showing first 100 items - show all)- Exact imposition of boundary conditions with distance functions in physics-informed deep neural networks
- CENN: conservative energy method based on neural networks with subdomains for solving variational problems involving heterogeneous and complex geometries
- NH-PINN: neural homogenization-based physics-informed neural network for multiscale problems
- Unified regression model in fitting potential energy surfaces for quantum dynamics
- Theory-guided physics-informed neural networks for boundary layer problems with singular perturbation
- A deep domain decomposition method based on Fourier features
- SPINN: sparse, physics-based, and partially interpretable neural networks for PDEs
- Solving and learning nonlinear PDEs with Gaussian processes
- Enforcing exact physics in scientific machine learning: a data-driven exterior calculus on graphs
- When and why PINNs fail to train: a neural tangent kernel perspective
- Multi-variance replica exchange SGMCMC for inverse and forward problems via Bayesian PINN
- CAN-PINN: a fast physics-informed neural network based on coupled-automatic-numerical differentiation method
- Physics-informed neural network simulation of multiphase poroelasticity using stress-split sequential training
- Improved architectures and training algorithms for deep operator networks
- A composite neural network that learns from multi-fidelity data: application to function approximation and inverse PDE problems
- ModalPINN: an extension of physics-informed neural networks with enforced truncated Fourier decomposition for periodic flow reconstruction using a limited number of imperfect sensors
- Scalable uncertainty quantification for deep operator networks using randomized priors
- Improved deep neural networks with domain decomposition in solving partial differential equations
- Physics-informed neural networks combined with polynomial interpolation to solve nonlinear partial differential equations
- Long-time integration of parametric evolution equations with physics-informed DeepONets
- Transfer learning based physics-informed neural networks for solving inverse problems in engineering structures under different loading scenarios
- opPINN: physics-informed neural network with operator learning to approximate solutions to the Fokker-Planck-Landau equation
- An unsupervised latent/output physics-informed convolutional-LSTM network for solving partial differential equations using peridynamic differential operator
- Deep energy method in topology optimization applications
- An overview on deep learning-based approximation methods for partial differential equations
- When do extended physics-informed neural networks (XPINNs) improve generalization?
- Active neuron least squares: a training method for multivariate rectified neural networks
- MOD-Net: a machine learning approach via model-operator-data network for solving PDEs
- A unified scalable framework for causal sweeping strategies for physics-informed neural networks (PINNs) and their temporal decompositions
- Deep learning methods for partial differential equations and related parameter identification problems
- Enhanced physics‐informed neural networks for hyperelasticity
- BINN: a deep learning approach for computational mechanics problems based on boundary integral equations
- Physics-informed radial basis network (PIRBN): a local approximating neural network for solving nonlinear partial differential equations
- Pre-training strategy for solving evolution equations based on physics-informed neural networks
- HomPINNs: homotopy physics-informed neural networks for solving the inverse problems of nonlinear differential equations with multiple solutions
- Respecting causality for training physics-informed neural networks
- Efficient physics-informed neural networks using hash encoding
- Bayesian Deep Learning Framework for Uncertainty Quantification in Stochastic Partial Differential Equations
- Subspace decomposition based DNN algorithm for elliptic type multi-scale PDEs
- Exact Dirichlet boundary physics-informed neural network EPINN for solid mechanics
- Finite basis physics-informed neural networks (FBPINNs): a scalable domain decomposition approach for solving differential equations
- A Neural Network Approach for Homogenization of Multiscale Problems
- On the spectral bias of coupled frequency predictor-corrector triangular DNN: the convergence analysis
- Feature-adjacent multi-fidelity physics-informed machine learning for partial differential equations
- Gradient-enhanced physics-informed neural networks based on transfer learning for inverse problems of the variable coefficient differential equations
- Is the neural tangent kernel of PINNs deep learning general partial differential equations always convergent?
- Physics-informed ConvNet: learning physical field from a shallow neural network
- Physical informed neural networks with soft and hard boundary constraints for solving advection-diffusion equations using Fourier expansions
- Residual-based attention in physics-informed neural networks
- Optimization of physics-informed neural networks for solving the nolinear Schrödinger equation
- Solving a class of multi-scale elliptic PDEs by means of Fourier-based mixed physics informed neural networks
- Constraint free physics-informed machine learning for micromagnetic energy minimization
- Dynamically configured physics-informed neural network in topology optimization applications
- Neural-integrated meshfree (NIM) method: a differentiable programming-based hybrid solver for computational mechanics
- CEENs: causality-enforced evolutional networks for solving time-dependent partial differential equations
- Physics-informed parallel neural networks with self-adaptive loss weighting for the identification of continuous structural systems
- An introduction to programming physics-informed neural network-based computational solid mechanics
- Learning scattering waves via coupling physics-informed neural networks and their convergence analysis
- Mixed formulation of physics-informed neural networks for thermo-mechanically coupled systems and heterogeneous domains
- Adaptive sampling points based multi-scale residual network for solving partial differential equations
- Multilevel domain decomposition-based architectures for physics-informed neural networks
- Variational temporal convolutional networks for I-FENN thermoelasticity
- Advanced physics-informed neural networks for numerical approximation of the coupled Schrödinger-KdV equation
- Interpretable physics-encoded finite element network to handle concentration features and multi-material heterogeneity in hyperelasticity
- Binary structured physics-informed neural networks for solving equations with rapidly changing solutions
- A block-coordinate approach of multi-level optimization with an application to physics-informed neural networks
- MHDnet: physics-preserving learning for solving magnetohydrodynamics problems
- Loss jump during loss switch in solving PDEs with neural networks
- Neural dynamical operator: continuous spatial-temporal model with gradient-based and derivative-free optimization methods
- I-FENN for thermoelasticity based on physics-informed temporal convolutional network (PI-TCN)
- Kolmogorov-Arnold-informed neural network: a physics-informed deep learning framework for solving forward and inverse problems based on Kolmogorov-Arnold networks
- Time integration schemes based on neural networks for solving partial differential equations on coarse grids
- Adaptive deep density approximation for stochastic dynamical systems
- Multiprecision computing for multistage fractional physics-informed neural networks
- A hybrid physics-informed neural network based multiscale solver as a partial differential equation constrained optimization problem
- TSPINN: Thompson sampling-based adaptive training for physics-informed neural networks
- Least-squares enhanced physics-informed learning for singular and ill-posed partial differential equations
- An efficient hybrid Fourier KAN-SepONet solver for the high-frequency Helmholtz equation and convergence analysis
- JostONet: a neural operator architecture for solving the Jost solution and scattering coefficients of the Schrödinger spectral problem
- Forecasting of spatiotemporal nonlinear dynamic systems by physics-informed neural networks with ResNet blocks
- Physics-informed neural network for solving neutron transport equation with differential order transformation and Fourier feature
- A novel Fourier feature network for solving partial differential equations
- Solving partial differential equations with random feature models
- Solving engineering eigenvalue problems with neural networks using the Rayleigh quotient
- FENN: feature-enhanced neural network for solving partial differential equations involving fluid mechanics
- Self-adaptive weights based on balanced residual decay rate for physics-informed neural networks and deep operator networks
- Coaxial permanent magnetic coupling analysis based on physics-informed neural network with periodic fixed boundary conditions
- Tensor neural network-based machine learning method for elliptic multiscale problems
- High-order multi-scale neural network method for quasi-static thermo-mechanical problems of composite materials
- Singular layer physics-informed neural network method for convection-dominated boundary layer problems in two dimensions
- Optimizing the optimizer for physics-informed neural networks and Kolmogorov-Arnold networks
- Decomposition-enhanced parameterized physics-informed neural network for uncertainty-aware interior acoustic analysis
- Two-scale neural networks for partial differential equations with small parameters
- Implicit bias in understanding deep learning for solving PDEs beyond Ritz-Galerkin method
- U-WNO: U-Net enhanced wavelet neural operator for solving parametric partial differential equations
- Multiscale data-driven modeling for nonlinear dynamical system with external inputs in porous media
- Overview frequency principle/spectral bias in deep learning
- Enhancing dynamic CT image reconstruction with neural fields and optical flow
- Interpolating between BSDEs and PINNs: deep learning for elliptic and parabolic boundary value problems
- Novel approach to solving Schwarzschild black hole perturbation equations via physics informed neural networks
Describes a project that uses
Uses Software
This page was built for publication: On the eigenvector bias of Fourier feature networks: from regression to solving multi-scale PDEs with physics-informed neural networks
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2237440)
