ReLU deep neural networks and linear finite elements

DOI10.4208/JCM.1901-M2018-0160MaRDI QIDQ3385667zbMATH OpenOpenAlexWikidataFDO

Authors Juncai He, Lin Li, Chunyue Zheng, Jinchao Xu

Publication date 14 January 2021

Published in Journal of Computational Mathematics (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1807.03973

zbMATH Keywords

finite element methodpiecewise linear functiondeep neural network

Mathematics Subject Classification ID

Artificial neural networks and deep learning (68T07) Finite element, Rayleigh-Ritz and Galerkin methods for boundary value problems involving PDEs (65N30)

Abstract: In this paper, we investigate the relationship between deep neural networks (DNN) with rectified linear unit (ReLU) function as the activation function and continuous piecewise linear (CPWL) functions, especially CPWL functions from the simplicial linear finite element method (FEM). We first consider the special case of FEM. By exploring the DNN representation of its nodal basis functions, we present a ReLU DNN representation of CPWL in FEM. We theoretically establish that at least $2$ hidden layers are needed in a ReLU DNN to represent any linear finite element functions in $Omegasubseteqmathbb{R}^d$ when $dge2$. Consequently, for $d=2,3$ which are often encountered in scientific and engineering computing, the minimal number of two hidden layers are necessary and sufficient for any CPWL function to be represented by a ReLU DNN. Then we include a detailed account on how a general CPWL in $mathbb{R}^d$ can be represented by a ReLU DNN with at most $lceillo{g}_2(d+1)ceil$ hidden layers and we also give an estimation of the number of neurons in DNN that are needed in such a representation. Furthermore, using the relationship between DNN and FEM, we theoretically argue that a special class of DNN models with low bit-width are still expected to have an adequate representation power in applications. Finally, as a proof of concept, we present some numerical results for using ReLU DNNs to solve a two point boundary problem to demonstrate the potential of applying DNN for numerical solution of partial differential equations.

Recommendations

• Deep ReLU networks and high-order finite element methods
• Spline representation and redundancies of one-dimensional ReLU neural network models
• Constructive deep ReLU neural network approximation
• Optimal approximation of piecewise smooth functions using deep ReLU neural networks
• PowerNet: efficient representations of polynomials and smooth functions by deep neural networks with rectified power units

Cited in

(62)

• Neural Network Method for Integral Fractional Laplace Equations
• Randomized Newton's method for solving differential equations based on the neural network discretization
• Solving parametric partial differential equations with deep rectified quadratic unit neural networks
• PDE-Net 2.0: learning PDEs from data with a numeric-symbolic hybrid deep network
• A Variational Neural Network Approach for Glacier Modelling with Nonlinear Rheology
• Gauss Newton method for solving variational problems of PDEs with neural network discretizaitons
• A functional equation with polynomial solutions and application to neural networks
• MIM: a deep mixed residual method for solving high-order partial differential equations
• A Novel Deep Neural Network Algorithm for the Helmholtz Scattering Problem In the Unbounded Domain
• A Deep Learning Method for Elliptic Hemivariational Inequalities
• MOD-Net: a machine learning approach via model-operator-data network for solving PDEs
• Adaptive two-layer ReLU neural network. I: Best least-squares approximation
• Adaptive two-layer ReLU neural network. II: Ritz approximation to elliptic PDEs
• A convergent deep learning algorithm for approximation of polynomials
• Can neural networks learn finite elements?
• Towards Lower Bounds on the Depth of ReLU Neural Networks
• Dying ReLU and initialization: theory and numerical examples
• ReLU neural network Galerkin BEM
• Piecewise linear functions representable with infinite width shallow ReLU neural networks
• Convergence rate of DeepONets for learning operators arising from advection-diffusion equations
• Physically motivated structuring and optimization of neural networks for multi-physics modelling of solid oxide fuel cells
• De Rham compatible deep neural network FEM
• Randomized neural network with Petrov-Galerkin methods for solving linear and nonlinear partial differential equations
• An Augmented Lagrangian Deep Learning Method for Variational Problems with Essential Boundary Conditions
• Deep neural networks can stably solve high-dimensional, noisy, non-linear inverse problems
• A mesh-less, ray-based deep neural network method for the Helmholtz equation with high frequency
• Machine learning architectures for price formation models
• Approximation of nonlinear functionals using deep ReLU networks
• Physics-constrained data-driven variational method for discrepancy modeling
• ReLU deep neural networks from the hierarchical basis perspective
• Recurrent neural networks as optimal mesh refinement strategies
• Physics-informed graph neural network emulation of soft-tissue mechanics
• Approximation and generalization of DeepONets for learning operators arising from a class of singularly perturbed problems
• Better approximations of high dimensional smooth functions by deep neural networks with rectified power units
• Learning quantized neural nets by coarse gradient method for nonlinear classification
• Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
• Data-driven Whitney forms for structure-preserving control volume analysis
• Solving nonconvex energy minimization problems in martensitic phase transitions with a mesh-free deep learning approach
• Finite neuron method and convergence analysis
• Neural network approximation
• Physics-Informed Neural Networks for Solving Dynamic Two-Phase Interface Problems
• DeepParticle: learning invariant measure by a deep neural network minimizing Wasserstein distance on data generated from an interacting particle method
• A theoretical analysis of deep neural networks and parametric PDEs
• Blended coarse gradient descent for full quantization of deep neural networks
• How can deep neural networks fail even with global optima?
• A machine learning approach to portfolio pricing and risk management for high‐dimensional problems
• A deep first-order system least squares method for solving elliptic PDEs
• Sparse Deep Neural Network for Nonlinear Partial Differential Equations
• Approximation properties of deep ReLU CNNs
• Adaptive quadratures for nonlinear approximation of low-dimensional PDEs using smooth neural networks
• SPINN: sparse, physics-based, and partially interpretable neural networks for PDEs
• Stable parameterization of continuous and piecewise-linear functions
• Simultaneous neural network approximation for smooth functions
• Applied harmonic analysis and data science. Abstracts from the workshop held November 28 -- December 4, 2021 (hybrid meeting)
• Estimating adsorption isotherm parameters in chromatography via a virtual injection promoting double feed-forward neural network
• Mini-workshop: Mathematics of entropic AI in the natural sciences. Abstracts from the mini-workshop held April 7--12, 2024
• A fast and accurate physics-informed neural network reduced order model with shallow masked autoencoder
• Galerkin neural networks: a framework for approximating variational equations with error control
• Multi-scale deep neural network (MscaleDNN) for solving Poisson-Boltzmann equation in complex domains
• Exponential ReLU neural network approximation rates for point and edge singularities
• Efficient coupled deep neural networks for the time-dependent coupled Stokes-Darcy problems
• Transport and scale interactions in geophysical flows. Abstracts from the workshop held July 16--21, 2023

This page was built for publication: ReLU deep neural networks and linear finite elements

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q3385667)