Optimal approximation of piecewise smooth functions using deep ReLU neural networks

DOI10.1016/J.NEUNET.2018.08.019MaRDI QIDQ2182898zbMATH OpenOpenAlexWikidataFDO

Authors Philipp Petersen, Felix Voigtlaender

Publication date 26 May 2020

Published in Neural Networks (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1709.05289

zbMATH Keywords

curse of dimensiondeep neural networksfunction approximationmetric entropypiecewise smooth functionssparse connectivity

Mathematics Subject Classification ID

Algorithms for approximation of functions (65D15) Artificial neural networks and deep learning (68T07)

Abstract: We study the necessary and sufficient complexity of ReLU neural networks---in terms of depth and number of weights---which is required for approximating classifier functions in $L^2$. As a model class, we consider the set $$ of possibly discontinuous piecewise $$ functions $f:[-1/2,1/2{]}^{d}omathbbR$, where the different smooth regions of $f$ are separated by $$ hypersurfaces. For dimension $dgeq2$, regularity $$, and accuracy $varepsilon>0$, we construct artificial neural networks with ReLU activation function that approximate functions from $$ up to $L^2$ error of $varepsilon$. The constructed networks have a fixed number of layers, depending only on $d$ and $$, and they have $$ many nonzero weights, which we prove to be optimal. In addition to the optimality in terms of the number of weights, we show that in order to achieve the optimal approximation rate, one needs ReLU networks of a certain depth. Precisely, for piecewise $$ functions, this minimal depth is given---up to a multiplicative constant---by $$. Up to a log factor, our constructed networks match this bound. This partly explains the benefits of depth for ReLU networks by showing that deep networks are necessary to achieve efficient approximation of (piecewise) smooth functions. Finally, we analyze approximation in high-dimensional spaces where the function $f$ to be approximated can be factorized into a smooth dimension reducing feature map $au$ and classifier function $g$---defined on a low-dimensional feature space---as $f=gcircau$. We show that in this case the approximation rate depends only on the dimension of the feature space and not the input dimension.

Recommendations

• Error bounds for approximations with deep ReLU networks
• Approximation of smoothness classes by deep rectifier networks
• Optimal approximation rate of ReLU networks in terms of width and depth
• Deep network approximation characterized by number of neurons
• Deep ReLU networks and high-order finite element methods

Cites work

• A logical calculus of the ideas immanent in nervous activity
• A Mathematical Theory of Deep Convolutional Neural Networks for Feature Extraction
• An Introduction to Banach Space Theory
• Approximation and estimation bounds for artificial neural networks
• Approximation by superpositions of a sigmoidal function
• Compactly supported shearlets are optimally sparse
• Deep learning
• Entropies of several sets of real valued functions
• Error bounds for approximations with deep ReLU networks
• Group invariant scattering
• scientific article; zbMATH DE number 410743 (Why is no real title available?)
• scientific article; zbMATH DE number 3491650 (Why is no real title available?)
• scientific article; zbMATH DE number 3539473 (Why is no real title available?)
• scientific article; zbMATH DE number 1249699 (Why is no real title available?)
• scientific article; zbMATH DE number 1324223 (Why is no real title available?)
• scientific article; zbMATH DE number 1022519 (Why is no real title available?)
• scientific article; zbMATH DE number 204193 (Why is no real title available?)
• scientific article; zbMATH DE number 1405266 (Why is no real title available?)
• scientific article; zbMATH DE number 3231758 (Why is no real title available?)
• Introduction to shearlets
• Introduction to Smooth Manifolds
• Lower bounds for approximation by MLP neural networks
• Multilayer feedforward networks are universal approximators
• New tight frames of curvelets and optimal representations of objects with piecewise C² singularities
• On the extension of a vector function so as to preserve a Lipschitz condition
• Optimally sparse data representations
• Optimally Sparse Multidimensional Representation Using Shearlets
• Real Analysis and Probability
• Representation and Compression of Multidimensional Piecewise Functions Using Surflets
• Sparse components of images and optimal atomic decompositions
• Unconditional bases are optimal bases for data compression and for statistical estimation
• Universal approximation bounds for superpositions of a sigmoidal function

Cited in

(only showing first 100 items - show all)

• Applied harmonic analysis and data processing. Abstracts from the workshop held March 25--31, 2018
• Provable approximation properties for deep neural networks
• Asymptotic expansion for neural network operators of the Kantorovich type and high order of approximation
• Efficient approximation of solutions of parametric linear transport equations by ReLU DNNs
• Topological properties of the set of functions generated by neural networks of fixed size
• Numerical solution of the parametric diffusion equation by deep neural networks
• Error bounds for deep ReLU networks using the Kolmogorov-Arnold superposition theorem
• Fast generalization error bound of deep learning without scale invariance of activation functions
• Approximation rates for neural networks with encodable weights in smoothness spaces
• Theory of deep convolutional neural networks. II: Spherical analysis
• High-dimensional distribution generation through deep neural networks
• Optimal approximation rate of ReLU networks in terms of width and depth
• Constructive deep ReLU neural network approximation
• Sparsest piecewise-linear regression of one-dimensional data
• Discontinuous neural networks and discontinuity learning
• On the approximation of rough functions with deep neural networks
• Adaptive machine learning-based surrogate modeling to accelerate PDE-constrained optimization in enhanced oil recovery
• CAS4DL: Christoffel adaptive sampling for function approximation via deep learning
• Stable recovery of entangled weights: towards robust identification of deep neural networks from minimal samples
• A measure theoretical approach to the mean-field maximum principle for training NeurODEs
• DNN expression rate analysis of high-dimensional PDEs: application to option pricing
• A theoretical analysis of deep neural networks and parametric PDEs
• Neural network identifiability for a family of sigmoidal nonlinearities
• Approximation spaces of deep neural networks
• Exponential ReLU DNN expression of holomorphic maps in high dimension
• Best \(k\)-layer neural network approximations
• High-order approximation rates for shallow neural networks with cosine and \(\mathrm{ReLU}^k\) activation functions
• SelectNet: self-paced learning for high-dimensional partial differential equations
• Multivariate neural network interpolation operators
• Solving eigenvalue PDEs of metastable diffusion processes using artificial neural networks
• Nonlinear approximation via compositions
• On the minimax optimality and superiority of deep neural network learning over sparse parameter spaces
• Theory of deep convolutional neural networks: downsampling
• Affine symmetries and neural network identifiability
• Approximating smooth functions by deep neural networks with sigmoid activation function
• Deep ReLU network expression rates for option prices in high-dimensional, exponential Lévy models
• Universality of deep convolutional neural networks
• Machine learning approximation algorithms for high-dimensional fully nonlinear partial differential equations and second-order backward stochastic differential equations
• Space-time error estimates for deep neural network approximations for differential equations
• A convenient infinite dimensional framework for generative adversarial learning
• Phase transitions in rate distortion theory and deep learning
• The universal approximation theorem for complex-valued neural networks
• Computation and learning in high dimensions. Abstracts from the workshop held August 1--7, 2021 (hybrid meeting)
• Approximation properties of residual neural networks for Kolmogorov PDEs
• Learning elliptic partial differential equations with randomized linear algebra
• ReLU deep neural networks and linear finite elements
• The gap between theory and practice in function approximation with deep neural networks
• New Function Spaces Associated to Representations of Nilpotent Lie Groups and Generalized Time-Frequency Analysis
• Deep network with approximation error being reciprocal of width to power of square root of depth
• Approximation of smoothness classes by deep rectifier networks
• Optimal approximation with sparsely connected deep neural networks
• New error bounds for deep ReLU networks using sparse grids
• Analysis of the generalization error: empirical risk minimization over deep artificial neural networks overcomes the curse of dimensionality in the numerical approximation of Black-Scholes partial differential equations
• scientific article; zbMATH DE number 7626714 (Why is no real title available?)
• Discretization of parameter identification in PDEs using neural networks
• A deep learning approach to Reduced Order Modelling of parameter dependent partial differential equations
• Theoretical issues in deep networks
• Quantitative Approximation Results for Complex-Valued Neural Networks
• Approximations with deep neural networks in Sobolev time-space
• Deep ReLU Networks Overcome the Curse of Dimensionality for Generalized Bandlimited Functions
• Full error analysis for the training of deep neural networks
• Neural parametric Fokker-Planck equation
• Deep neural network surrogates for nonsmooth quantities of interest in shape uncertainty quantification
• Deep ReLU networks and high-order finite element methods
• Error bounds for approximations with deep ReLU neural networks in \(W^{s , p}\) norms
• Rectified deep neural networks overcome the curse of dimensionality for nonsmooth value functions in zero-sum games of nonlinear stiff systems
• Deep neural network expression of posterior expectations in Bayesian PDE inversion
• Adaptive approximation and generalization of deep neural network with intrinsic dimensionality
• Topology of deep neural networks
• Deep Network Approximation for Smooth Functions
• Plateau phenomenon in gradient descent training of RELU networks: explanation, quantification, and avoidance
• Better approximations of high dimensional smooth functions by deep neural networks with rectified power units
• Dying ReLU and initialization: theory and numerical examples
• Deep network approximation characterized by number of neurons
• Randomized Optimal Stopping Algorithms and Their Convergence Analysis
• Equivalence of approximation by convolutional neural networks and fully-connected networks
• Deep ReLU neural networks overcome the curse of dimensionality for partial integrodifferential equations
• Deep neural networks can stably solve high-dimensional, noisy, non-linear inverse problems
• Spline representation and redundancies of one-dimensional ReLU neural network models
• scientific article; zbMATH DE number 7660136 (Why is no real title available?)
• Neural network approximation
• A Proof that Artificial Neural Networks Overcome the Curse of Dimensionality in the Numerical Approximation of Black–Scholes Partial Differential Equations
• Approximation bounds for norm constrained neural networks with applications to regression and GANs
• Solving second-order nonlinear evolution partial differential equations using deep learning
• Simultaneous neural network approximation for smooth functions
• Deep Neural Networks for Solving Large Linear Systems Arising from High-Dimensional Problems
• Neural network approximation: three hidden layers are enough
• A deep network construction that adapts to intrinsic dimensionality beyond the domain
• Theory of deep convolutional neural networks. III: Approximating radial functions
• Approximation capabilities of neural networks on unbounded domains
• On the capacity of deep generative networks for approximating distributions
• De Rham compatible deep neural network FEM
• Deep ReLU neural network approximation in Bochner spaces and applications to parametric PDEs
• Approximating smooth and sparse functions by deep neural networks: optimal approximation rates and saturation
• Rates of approximation by ReLU shallow neural networks
• Neural Network Approximation of Coarse-Scale Surrogates in Numerical Homogenization
• Three ways to solve partial differential equations with neural networks — A review
• Deep learning methods for partial differential equations and related parameter identification problems
• Fast convergence rates of deep neural networks for classification
• On decision regions of narrow deep neural networks

Describes a project that uses

Uses Software

• AlexNet
• ImageNet

This page was built for publication: Optimal approximation of piecewise smooth functions using deep ReLU neural networks

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2182898)