ReLU deep neural networks and linear finite elements

DOI10.4208/JCM.1901-M2018-0160zbMATH Open1463.68072arXiv1807.03973OpenAlexW3099849883WikidataQ114021207 ScholiaQ114021207MaRDI QIDQ3385667FDOQ3385667

Authors: Juncai He, Lin Li, Chunyue Zheng, Jinchao Xu 

Publication date: 14 January 2021

Published in: Journal of Computational Mathematics (Search for Journal in Brave)

Abstract: In this paper, we investigate the relationship between deep neural networks (DNN) with rectified linear unit (ReLU) function as the activation function and continuous piecewise linear (CPWL) functions, especially CPWL functions from the simplicial linear finite element method (FEM). We first consider the special case of FEM. By exploring the DNN representation of its nodal basis functions, we present a ReLU DNN representation of CPWL in FEM. We theoretically establish that at least $2$ hidden layers are needed in a ReLU DNN to represent any linear finite element functions in $Omegasubseteqmathbb{R}^d$ when $dge2$. Consequently, for $d=2,3$ which are often encountered in scientific and engineering computing, the minimal number of two hidden layers are necessary and sufficient for any CPWL function to be represented by a ReLU DNN. Then we include a detailed account on how a general CPWL in $mathbb{R}^d$ can be represented by a ReLU DNN with at most $lceillo{g}_2(d+1)ceil$ hidden layers and we also give an estimation of the number of neurons in DNN that are needed in such a representation. Furthermore, using the relationship between DNN and FEM, we theoretically argue that a special class of DNN models with low bit-width are still expected to have an adequate representation power in applications. Finally, as a proof of concept, we present some numerical results for using ReLU DNNs to solve a two point boundary problem to demonstrate the potential of applying DNN for numerical solution of partial differential equations.

Full work available at URL: https://arxiv.org/abs/1807.03973

Recommendations

• Deep ReLU networks and high-order finite element methods
• Spline representation and redundancies of one-dimensional ReLU neural network models
• Constructive deep ReLU neural network approximation
• Optimal approximation of piecewise smooth functions using deep ReLU neural networks
• PowerNet: efficient representations of polynomials and smooth functions by deep neural networks with rectified power units

zbMATH Keywords

finite element methodpiecewise linear functiondeep neural network

Mathematics Subject Classification ID

Artificial neural networks and deep learning (68T07) Finite element, Rayleigh-Ritz and Galerkin methods for boundary value problems involving PDEs (65N30)

Cited In (62)

• Gauss Newton method for solving variational problems of PDEs with neural network discretizaitons
• Can neural networks learn finite elements?
• Physics-constrained data-driven variational method for discrepancy modeling
• Physics-informed graph neural network emulation of soft-tissue mechanics
• Approximation and generalization of DeepONets for learning operators arising from a class of singularly perturbed problems
• Data-driven Whitney forms for structure-preserving control volume analysis
• How can deep neural networks fail even with global optima?
• Adaptive quadratures for nonlinear approximation of low-dimensional PDEs using smooth neural networks
• Mini-workshop: Mathematics of entropic AI in the natural sciences. Abstracts from the mini-workshop held April 7--12, 2024
• Transport and scale interactions in geophysical flows. Abstracts from the workshop held July 16--21, 2023
• Randomized Newton's method for solving differential equations based on the neural network discretization
• Solving parametric partial differential equations with deep rectified quadratic unit neural networks
• A Variational Neural Network Approach for Glacier Modelling with Nonlinear Rheology
• PDE-Net 2.0: learning PDEs from data with a numeric-symbolic hybrid deep network
• A Novel Deep Neural Network Algorithm for the Helmholtz Scattering Problem In the Unbounded Domain
• A functional equation with polynomial solutions and application to neural networks
• MIM: a deep mixed residual method for solving high-order partial differential equations
• A Deep Learning Method for Elliptic Hemivariational Inequalities
• MOD-Net: a machine learning approach via model-operator-data network for solving PDEs
• Adaptive two-layer ReLU neural network. I: Best least-squares approximation
• Adaptive two-layer ReLU neural network. II: Ritz approximation to elliptic PDEs
• A convergent deep learning algorithm for approximation of polynomials
• Towards Lower Bounds on the Depth of ReLU Neural Networks
• Dying ReLU and initialization: theory and numerical examples
• ReLU neural network Galerkin BEM
• Piecewise linear functions representable with infinite width shallow ReLU neural networks
• Convergence rate of DeepONets for learning operators arising from advection-diffusion equations
• Physically motivated structuring and optimization of neural networks for multi-physics modelling of solid oxide fuel cells
• De Rham compatible deep neural network FEM
• Randomized neural network with Petrov-Galerkin methods for solving linear and nonlinear partial differential equations
• An Augmented Lagrangian Deep Learning Method for Variational Problems with Essential Boundary Conditions
• Deep neural networks can stably solve high-dimensional, noisy, non-linear inverse problems
• Machine learning architectures for price formation models
• A mesh-less, ray-based deep neural network method for the Helmholtz equation with high frequency
• Approximation of nonlinear functionals using deep ReLU networks
• ReLU deep neural networks from the hierarchical basis perspective
• Recurrent neural networks as optimal mesh refinement strategies
• Better approximations of high dimensional smooth functions by deep neural networks with rectified power units
• Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
• Learning quantized neural nets by coarse gradient method for nonlinear classification
• Solving nonconvex energy minimization problems in martensitic phase transitions with a mesh-free deep learning approach
• Neural network approximation
• Physics-Informed Neural Networks for Solving Dynamic Two-Phase Interface Problems
• Finite neuron method and convergence analysis
• DeepParticle: learning invariant measure by a deep neural network minimizing Wasserstein distance on data generated from an interacting particle method
• A theoretical analysis of deep neural networks and parametric PDEs
• Blended coarse gradient descent for full quantization of deep neural networks
• A machine learning approach to portfolio pricing and risk management for high‐dimensional problems
• Sparse Deep Neural Network for Nonlinear Partial Differential Equations
• A deep first-order system least squares method for solving elliptic PDEs
• Approximation properties of deep ReLU CNNs
• Stable parameterization of continuous and piecewise-linear functions
• Simultaneous neural network approximation for smooth functions
• SPINN: sparse, physics-based, and partially interpretable neural networks for PDEs
• Applied harmonic analysis and data science. Abstracts from the workshop held November 28 -- December 4, 2021 (hybrid meeting)
• Estimating adsorption isotherm parameters in chromatography via a virtual injection promoting double feed-forward neural network
• A fast and accurate physics-informed neural network reduced order model with shallow masked autoencoder
• Galerkin neural networks: a framework for approximating variational equations with error control
• Multi-scale deep neural network (MscaleDNN) for solving Poisson-Boltzmann equation in complex domains
• Exponential ReLU neural network approximation rates for point and edge singularities
• Efficient coupled deep neural networks for the time-dependent coupled Stokes-Darcy problems
• Neural Network Method for Integral Fractional Laplace Equations

This page was built for publication: ReLU deep neural networks and linear finite elements

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q3385667)