Connections between numerical algorithms for PDEs and neural networks
From MaRDI portal
Publication:6156049
Abstract: We investigate numerous structural connections between numerical algorithms for partial differential equations (PDEs) and neural architectures. Our goal is to transfer the rich set of mathematical foundations from the world of PDEs to neural networks. Besides structural insights we provide concrete examples and experimental evaluations of the resulting architectures. Using the example of generalised nonlinear diffusion in 1D, we consider explicit schemes, acceleration strategies thereof, implicit schemes, and multigrid approaches. We connect these concepts to residual networks, recurrent neural networks, and U-net architectures. Our findings inspire a symmetric residual network design with provable stability guarantees and justify the effectiveness of skip connections in neural networks from a numerical perspective. Moreover, we present U-net architectures that implement multigrid techniques for learning efficient solutions of partial differential equation models, and motivate uncommon design choices such as trainable nonmonotone activation functions. Experimental evaluations show that the proposed architectures save half of the trainable parameters and can thus outperform standard ones with the same model complexity. Our considerations serve as a basis for explaining the success of popular neural architectures and provide a blueprint for developing new mathematically well-founded neural building blocks.
Recommendations
- Neural‐network‐based approximations for solving partial differential equations
- Neural networks and numerical analysis
- scientific article; zbMATH DE number 1466140
- Translating numerical concepts for PDEs into neural architectures
- scientific article; zbMATH DE number 957047
- scientific article; zbMATH DE number 671791
- Neural network method for solving partial differential equations
- An introduction to neural network methods for differential equations
- Solving parametric PDE problems with artificial neural networks
Cites work
- scientific article; zbMATH DE number 3958638 (Why is no real title available?)
- scientific article; zbMATH DE number 1111402 (Why is no real title available?)
- scientific article; zbMATH DE number 1953444 (Why is no real title available?)
- scientific article; zbMATH DE number 202526 (Why is no real title available?)
- scientific article; zbMATH DE number 3085412 (Why is no real title available?)
- A Multigrid Tutorial, Second Edition
- A multigrid platform for real-time motion computation with discontinuity-preserving variational methods
- A neural network multigrid solver for the Navier-Stokes equations
- A theoretical analysis of deep neural networks and parametric PDEs
- Another Look at Neural Multigrid
- Black-box learning of multigrid parameters
- CLIP: cheap Lipschitz training of neural networks
- Deep learning
- Deep neural network structures solving variational inequalities
- Deep neural networks motivated by partial differential equations
- Equivariant deep learning via morphological and linear scale space PDEs on the space of positions and orientations
- Forward stability of ResNet and its variants
- Fourth-order partial differential equations for noise removal
- Generalized Du Fort–Frankel Methods for Parabolic Initial-Boundary Value Problems
- Image Processing and Analysis
- Image compression with anisotropic diffusion
- Layer-Parallel Training of Deep Residual Neural Networks
- Learning partial differential equations via data discovery and sparse optimization
- Lipschitz Certificates for Layered Network Structures Driven by Averaged Activation Operators
- MgNet: a unified framework of multigrid and convolutional neural network
- Minimizing total variation flow
- Multi-Level Adaptive Solutions to Boundary-Value Problems
- Multigrid algorithm for high order denoising
- Networks for nonlinear diffusion problems in imaging
- Neural networks and physical systems with emergent collective computational abilities
- Nonlinear approximation and (deep) ReLU networks
- Nonlinear total variation based noise removal algorithms
- On Lipschitz Bounds of General Convolutional Neural Networks
- On the Internal Stability of Explicit,m-Stage Runge-Kutta Methods for Largem-Values
- Optimal signalling in attractor neural networks
- PDE-Net 2.0: learning PDEs from data with a numeric-symbolic hybrid deep network
- Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
- Properties of higher order nonlinear diffusion filtering
- Regularisation of neural networks by enforcing Lipschitz continuity
- Relations between regularization and diffusion filtering
- Residual networks as flows of diffeomorphisms
- Some methods of speeding up the convergence of iteration methods
- Spectral theory of block operator matrices and applications
- Stable architectures for deep neural networks
- Translating numerical concepts for PDEs into neural architectures
- Visualization and processing of tensor fields.
- What Kinds of Functions Do Deep Neural Networks Learn? Insights from Variational Spline Theory
Cited in
(3)
This page was built for publication: Connections between numerical algorithms for PDEs and neural networks
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6156049)
