Connections between numerical algorithms for PDEs and neural networks

DOI10.1007/S10851-022-01106-XarXiv2107.14742OpenAlexW3187790213WikidataQ114226029 ScholiaQ114226029MaRDI QIDQ6156049FDOQ6156049

Authors: Tobias Alt, Karl Schrader, Matthias Augustin, Pascal Peter, Joachim Weickert 

Publication date: 12 June 2023

Published in: Journal of Mathematical Imaging and Vision (Search for Journal in Brave)

Abstract: We investigate numerous structural connections between numerical algorithms for partial differential equations (PDEs) and neural architectures. Our goal is to transfer the rich set of mathematical foundations from the world of PDEs to neural networks. Besides structural insights we provide concrete examples and experimental evaluations of the resulting architectures. Using the example of generalised nonlinear diffusion in 1D, we consider explicit schemes, acceleration strategies thereof, implicit schemes, and multigrid approaches. We connect these concepts to residual networks, recurrent neural networks, and U-net architectures. Our findings inspire a symmetric residual network design with provable stability guarantees and justify the effectiveness of skip connections in neural networks from a numerical perspective. Moreover, we present U-net architectures that implement multigrid techniques for learning efficient solutions of partial differential equation models, and motivate uncommon design choices such as trainable nonmonotone activation functions. Experimental evaluations show that the proposed architectures save half of the trainable parameters and can thus outperform standard ones with the same model complexity. Our considerations serve as a basis for explaining the success of popular neural architectures and provide a blueprint for developing new mathematically well-founded neural building blocks.

Full work available at URL: https://arxiv.org/abs/2107.14742

Recommendations

• Neural‐network‐based approximations for solving partial differential equations
• Neural networks and numerical analysis
• scientific article; zbMATH DE number 1466140
• Translating numerical concepts for PDEs into neural architectures
• scientific article; zbMATH DE number 957047
• scientific article; zbMATH DE number 671791
• Neural network method for solving partial differential equations
• An introduction to neural network methods for differential equations
• Solving parametric PDE problems with artificial neural networks

zbMATH Keywords

stabilityneural networksnonlinear diffusionpartial differential equationsnumerical algorithms

Mathematics Subject Classification ID

Computer science (68-XX) Information and communication theory, circuits (94-XX)

Cites Work

• Nonlinear total variation based noise removal algorithms
• PDE-Net 2.0: learning PDEs from data with a numeric-symbolic hybrid deep network
• MgNet: a unified framework of multigrid and convolutional neural network
• Title not available (Why is that?)
• Image Processing and Analysis
• Deep learning
• Fourth-order partial differential equations for noise removal
• Title not available (Why is that?)
• Neural networks and physical systems with emergent collective computational abilities
• Multi-Level Adaptive Solutions to Boundary-Value Problems
• Title not available (Why is that?)
• Spectral theory of block operator matrices and applications
• A Multigrid Tutorial, Second Edition
• Minimizing total variation flow
• Relations between regularization and diffusion filtering
• On the Internal Stability of Explicit,m-Stage Runge-Kutta Methods for Largem-Values
• Some methods of speeding up the convergence of iteration methods
• Visualization and processing of tensor fields.
• Multigrid algorithm for high order denoising
• Title not available (Why is that?)
• A multigrid platform for real-time motion computation with discontinuity-preserving variational methods
• Properties of higher order nonlinear diffusion filtering
• Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
• Forward stability of ResNet and its variants
• Deep neural networks motivated by partial differential equations
• Residual networks as flows of diffeomorphisms
• Regularisation of neural networks by enforcing Lipschitz continuity
• Deep neural network structures solving variational inequalities
• Image compression with anisotropic diffusion
• Optimal signalling in attractor neural networks
• Stable architectures for deep neural networks
• Title not available (Why is that?)
• Lipschitz Certificates for Layered Network Structures Driven by Averaged Activation Operators
• On Lipschitz Bounds of General Convolutional Neural Networks
• Equivariant deep learning via morphological and linear scale space PDEs on the space of positions and orientations
• Translating numerical concepts for PDEs into neural architectures
• CLIP: cheap Lipschitz training of neural networks
• Generalized Du Fort–Frankel Methods for Parabolic Initial-Boundary Value Problems
• Networks for nonlinear diffusion problems in imaging
• Learning partial differential equations via data discovery and sparse optimization
• Layer-Parallel Training of Deep Residual Neural Networks
• A neural network multigrid solver for the Navier-Stokes equations
• Nonlinear approximation and (deep) ReLU networks
• A theoretical analysis of deep neural networks and parametric PDEs
• Black-box learning of multigrid parameters
• What Kinds of Functions Do Deep Neural Networks Learn? Insights from Variational Spline Theory
• Another Look at Neural Multigrid

Cited In (3)

• Learning phase field mean curvature flows with neural networks
• Comparison of neural closure models for discretised PDEs
• Greedy training algorithms for neural networks and applications to PDEs

This page was built for publication: Connections between numerical algorithms for PDEs and neural networks

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6156049)