Optimal approximation rate of ReLU networks in terms of width and depth

DOI10.1016/J.MATPUR.2021.07.009MaRDI QIDQ2065073zbMATH OpenOpenAlexFDO

Authors Zuowei Shen, Haizhao Yang, Shijun Zhang

Publication date 7 January 2022

Published in Journal de Mathématiques Pures et Appliquées. Neuvième Série (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2103.00502

zbMATH Keywords

bit extractiondeep ReLU networksoptimal approximationVC-dimension

Mathematics Subject Classification ID

Rate of convergence, degree of approximation (41A25) Approximation by arbitrary nonlinear expressions; widths and entropy (41A46) Multidimensional problems (41A63) Artificial neural networks and deep learning (68T07)

Abstract: This paper concentrates on the approximation power of deep feed-forward neural networks in terms of width and depth. It is proved by construction that ReLU networks with width $$ and depth $mathcalO(L)$ can approximate a H"older continuous function on $[0,1{]}^d$ with an approximation rate $$, where $alphain(0,1]$ and $lambda>0$ are H"older order and constant, respectively. Such a rate is optimal up to a constant in terms of width and depth separately, while existing results are only nearly optimal without the logarithmic factor in the approximation rate. More generally, for an arbitrary continuous function $f$ on $[0,1{]}^d$, the approximation rate becomes $$, where $omeg{a}_f(cdot)$ is the modulus of continuity. We also extend our analysis to any continuous function $f$ on a bounded set. Particularly, if ReLU networks with depth $31$ and width $mathcalO(N)$ are used to approximate one-dimensional Lipschitz continuous functions on $[0,1]$ with a Lipschitz constant $lambda>0$, the approximation rate in terms of the total number of parameters, $W=mathcalO(N^2)$, becomes $mathcalO(fraclambdaWlnW)$, which has not been discovered in the literature for fixed-depth ReLU networks.

Recommendations

• Deep network approximation characterized by number of neurons
• Optimal approximation of piecewise smooth functions using deep ReLU neural networks
• Error bounds for approximations with deep ReLU networks
• On sharpness of an error bound for deep ReLU network approximation
• Deep network with approximation error being reciprocal of width to power of square root of depth

Cites work

• A note on the expressive power of deep rectified linear unit networks in high-dimensional spaces
• A priori estimates of the population risk for two-layer neural networks
• Approximation by superpositions of a sigmoidal function
• Approximation rates for neural networks with general activation functions
• Deep network approximation characterized by number of neurons
• Deep network with approximation error being reciprocal of width to power of square root of depth
• Deep ReLU Networks Overcome the Curse of Dimensionality for Generalized Bandlimited Functions
• Efficient distribution-free learning of probabilistic concepts
• Error bounds for approximations with deep ReLU networks
• Exponential convergence of the deep neural network approximation for analytic functions
• Make _1 regularization effective in training sparse CNN
• Multilayer feedforward networks are universal approximators
• Neural network approximation: three hidden layers are enough
• Nonlinear approximation via compositions
• Optimal approximation of piecewise smooth functions using deep ReLU neural networks
• Stochastic modified equations and dynamics of stochastic gradient algorithms. I: Mathematical foundations
• Universal approximation bounds for superpositions of a sigmoidal function

Cited in

(47)

• Approximation results on nonlinear operators by \(P_p\)-statistical convergence
• Error bounds for ReLU networks with depth and width parameters
• On sharpness of an error bound for deep ReLU network approximation
• Optimal approximation of piecewise smooth functions using deep ReLU neural networks
• Active learning based sampling for high-dimensional nonlinear partial differential equations
• Side effects of learning from low-dimensional data embedded in a Euclidean space
• Greedy training algorithms for neural networks and applications to PDEs
• Deep network with approximation error being reciprocal of width to power of square root of depth
• DENSITY RESULTS BY DEEP NEURAL NETWORK OPERATORS WITH INTEGER WEIGHTS
• Deep Network Approximation for Smooth Functions
• Deep network approximation characterized by number of neurons
• Rates of approximation by ReLU shallow neural networks
• ReLU neural networks of polynomial size for exact maximum flow computation
• Towards Lower Bounds on the Depth of ReLU Neural Networks
• Universal regular conditional distributions via probabilistic transformers
• Deep Neural Networks with ReLU-Sine-Exponential Activations Break Curse of Dimensionality in Approximation on Hölder Class
• Deep nonparametric regression on approximate manifolds: nonasymptotic error bounds with polynomial prefactors
• Deep learning via dynamical systems: an approximation perspective
• Solving PDEs on unknown manifolds with machine learning
• Low dimensional approximation and generalization of multivariate functions on smooth manifolds using deep ReLU neural networks
• Gauss Newton method for solving variational problems of PDEs with neural network discretizaitons
• How do noise tails impact on deep ReLU networks?
• Computing ground states of Bose-Einstein condensation by normalized deep neural network
• Weighted variation spaces and approximation by shallow ReLU networks
• Mini-workshop: Mathematics of entropic AI in the natural sciences. Abstracts from the mini-workshop held April 7--12, 2024
• Convergence and error control of consistent PINNs for elliptic PDEs
• Data-driven policy learning for continuous treatments
• Deep Regression for Repeated Measurements
• On non-asymptotic theory of recurrent neural networks in temporal point processes
• DRM revisited: a complete error analysis
• On the statistical properties of generative adversarial models for low intrinsic data dimension
• Sharp lower bounds on interpolation by deep ReLU neural networks at irregularly spaced data
• On the rates of convergence for learning with convolutional neural networks
• Structured and balanced multicomponent and multilayer neural networks
• Neural networks meet random forests
• On the optimal approximation of Sobolev and Besov functions using deep ReLU neural networks
• A hybrid iterative method for elliptic variational inequalities of the second kind
• Compositional sparsity, approximation classes, and parametric transport equations
• Deep neural networks for nonparametric interaction models with diverging dimension
• Model Selection for Multivalued-Treatment Policy Learning in Observational Studies
• Approximation of Analytic Functions by ReLU Neural Networks with Adjustable Depth and Width
• Approximation of solutions of parameter-dependent problems by residual neural networks
• Curse of dimensionality in neural network optimization
• Barron space for graph convolutional neural networks
• Approximation and error bounds for Kolmogorov-Arnold networks applications to dynamical system discovery via linear multistep methods
• Stable learning using spiking neural networks equipped with affine encoders and decoders
• Neural operators can play dynamic Stackelberg games

This page was built for publication: Optimal approximation rate of ReLU networks in terms of width and depth

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2065073)