Simultaneous neural network approximation for smooth functions

DOI10.1016/J.NEUNET.2022.06.040MaRDI QIDQ6052416zbMATH OpenWikidataFDO

Authors Sean Y. Hon, Haizhao Yang

Publication date 17 October 2023

Published in Neural Networks (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2109.00161

zbMATH Keywords

approximation theorydeep neural networksReLU\(^k\) activation functionsSobolev norm

Mathematics Subject Classification ID

Rate of convergence, degree of approximation (41A25) Artificial neural networks and deep learning (68T07)

Abstract: We establish in this work approximation results of deep neural networks for smooth functions measured in Sobolev norms, motivated by recent development of numerical solvers for partial differential equations using deep neural networks. {Our approximation results are nonasymptotic in the sense that the error bounds are explicitly characterized in terms of both the width and depth of the networks simultaneously with all involved constants explicitly determined.} Namely, for $fin{C}^s([0,1{]}^d)$, we show that deep ReLU networks of width $mathcalO(NlogN)$ and of depth $mathcalO(LlogL)$ can achieve a nonasymptotic approximation rate of $mathcalO(N^{-2(s-1)/d}{L}^{-2(s-1)/d})$ with respect to the $mathcal{W}^{1,p}([0,1{]}^d)$ norm for $pin[1,infty)$. If either the ReLU function or its square is applied as activation functions to construct deep neural networks of width $mathcalO(NlogN)$ and of depth $mathcalO(LlogL)$ to approximate $fin{C}^s([0,1{]}^d)$, the approximation rate is $mathcalO(N^{-2(s-n)/d}{L}^{-2(s-n)/d})$ with respect to the $mathcal{W}^{n,p}([0,1{]}^d)$ norm for $pin[1,infty)$.

Recommendations

• Deep Network Approximation for Smooth Functions
• Neural-network approximation of functions of several variables
• A simultaneous neural network approximation with the squashing function
• Approximating smooth functions by deep neural networks with sigmoid activation function
• scientific article; zbMATH DE number 5839712
• Simultaneous approximation of a smooth function and its derivatives by deep neural networks with piecewise-polynomial activations
• Neural networks for optimal approximation of continuous functions in \(\mathbb R^d\)

Cites work

• A mean field view of the landscape of two-layer neural networks
• A priori estimates of the population risk for two-layer neural networks
• A proposal on machine learning via dynamical systems
• Analysis of the generalization error: empirical risk minimization over deep artificial neural networks overcomes the curse of dimensionality in the numerical approximation of Black-Scholes partial differential equations
• Approximation rates for neural networks with encodable weights in smoothness spaces
• Approximation rates for neural networks with general activation functions
• Approximation spaces of deep neural networks
• Data driven governing equations approximation using deep neural networks
• Deep learning in high dimension: neural network expression rates for generalized polynomial chaos expansions in UQ
• Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations
• Deep network approximation characterized by number of neurons
• Deep Network Approximation for Smooth Functions
• Deep network with approximation error being reciprocal of width to power of square root of depth
• Deep ReLU networks and high-order finite element methods
• DNN expression rate analysis of high-dimensional PDEs: application to option pricing
• Error bounds for approximations with deep ReLU networks
• Error bounds for approximations with deep ReLU neural networks in \(W^{s , p}\) norms
• Error bounds for deep ReLU networks using the Kolmogorov-Arnold superposition theorem
• Exponential convergence of the deep neural network approximation for analytic functions
• Machine learning for prediction with missing dynamics
• Model reduction with memory and the machine learning of dynamical systems
• Neural network approximation
• Neural network approximation: three hidden layers are enough
• New error bounds for deep ReLU networks using sparse grids
• Nonlinear approximation and (deep) ReLU networks
• Nonlinear approximation via compositions
• Numerical solution of the parametric diffusion equation by deep neural networks
• Optimal approximation of piecewise smooth functions using deep ReLU neural networks
• Optimal approximation with sparsely connected deep neural networks
• Optimal nonlinear approximation
• PDE-Net 2.0: learning PDEs from data with a numeric-symbolic hybrid deep network
• Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
• ReLU deep neural networks and linear finite elements
• SelectNet: self-paced learning for high-dimensional partial differential equations
• Solving high-dimensional partial differential equations using deep learning
• Solving parametric PDE problems with artificial neural networks
• The Barron space and the flow-induced function spaces for neural network models
• Theoretical Insights Into the Optimization Landscape of Over-Parameterized Shallow Neural Networks

Cited in

(9)

• Approximating smooth functions by deep neural networks with sigmoid activation function
• Stationary Density Estimation of Itô Diffusions Using Deep Learning
• Friedrichs Learning: Weak Solutions of Partial Differential Equations via Deep Learning
• Simultaneous approximation of a smooth function and its derivatives by deep neural networks with piecewise-polynomial activations
• Computing ground states of Bose-Einstein condensation by normalized deep neural network
• Double-well net for image segmentation
• Simultaneous approximation by neural network operators with applications to Voronovskaja formulas
• Error analyses of the single hidden layer neural network based on the extreme learning machine for solving differential equations
• Approximation of Analytic Functions by ReLU Neural Networks with Adjustable Depth and Width

This page was built for publication: Simultaneous neural network approximation for smooth functions

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6052416)