Nonlinear approximation and (deep) ReLU networks
From MaRDI portal
Rate of convergence, degree of approximation (41A25) Approximation by other special function classes (41A30) Approximation by arbitrary nonlinear expressions; widths and entropy (41A46) Artificial neural networks and deep learning (68T07) Neural nets applied to problems in time-dependent statistical mechanics (82C32) Neural networks for/in biological studies, artificial life and related topics (92B20)
Abstract: This article is concerned with the approximation and expressive powers of deep neural networks. This is an active research area currently producing many interesting papers. The results most commonly found in the literature prove that neural networks approximate functions with classical smoothness to the same accuracy as classical linear methods of approximation, e.g. approximation by polynomials or by piecewise polynomials on prescribed partitions. However, approximation by neural networks depending on n parameters is a form of nonlinear approximation and as such should be compared with other nonlinear methods such as variable knot splines or n-term approximation from dictionaries. The performance of neural networks in targeted applications such as machine learning indicate that they actually possess even greater approximation power than these traditional methods of nonlinear approximation. The main results of this article prove that this is indeed the case. This is done by exhibiting large classes of functions which can be efficiently captured by neural networks where classical nonlinear methods fall short of the task. The present article purposefully limits itself to studying the approximation of univariate functions by ReLU networks. Many generalizations to functions of several variables and other activation functions can be envisioned. However, even in this simplest of settings considered here, a theory that completely quantifies the approximation power of neural networks is still lacking.
Recommendations
- Error bounds for approximations with deep ReLU networks
- A note on the expressive power of deep rectified linear unit networks in high-dimensional spaces
- Approximation spaces of deep neural networks
- Provable approximation properties for deep neural networks
- Deep ReLU networks and high-order finite element methods
Cites work
- Approximation by superpositions of a sigmoidal function
- Deep learning in high dimension: neural network expression rates for generalized polynomial chaos expansions in UQ
- Deep network approximation characterized by number of neurons
- Deep Network Approximation for Smooth Functions
- Deep vs. shallow networks: an approximation theory perspective
- Error bounds for approximations with deep ReLU networks
- Exponential convergence of the deep neural network approximation for analytic functions
- scientific article; zbMATH DE number 4075444 (Why is no real title available?)
- scientific article; zbMATH DE number 1215245 (Why is no real title available?)
- scientific article; zbMATH DE number 477682 (Why is no real title available?)
- scientific article; zbMATH DE number 1889798 (Why is no real title available?)
- Multilayer feedforward networks are universal approximators
- Neural Networks for Localized Approximation
- Optimal approximation with sparsely connected deep neural networks
- Optimal nonlinear approximation
- Provable approximation properties for deep neural networks
- The Takagi function: a survey
- Wavelet compression and nonlinear n-widths
- Weierstrass' function and chaos
Cited in
(only showing first 100 items - show all)- Linearized two-layers neural networks in high dimension
- High-dimensional distribution generation through deep neural networks
- Constructive deep ReLU neural network approximation
- The construction and approximation of ReLU neural network operators
- On the approximation of rough functions with deep neural networks
- Why rectified linear activation functions? Why max-pooling? A possible explanation
- Stable recovery of entangled weights: towards robust identification of deep neural networks from minimal samples
- Information theory and recovery algorithms for data fusion in Earth observation
- Depth separations in neural networks: what is actually being separated?
- Approximation spaces of deep neural networks
- Exponential ReLU DNN expression of holomorphic maps in high dimension
- Adaptive two-layer ReLU neural network. I: Best least-squares approximation
- Machine learning design of volume of fluid schemes for compressible flows
- Thermodynamically consistent physics-informed neural networks for hyperbolic systems
- Convergence rates of deep ReLU networks for multiclass classification
- A mesh-free method using piecewise deep neural network for elliptic interface problems
- Approximation properties of deep ReLU CNNs
- ReLU deep neural networks from the hierarchical basis perspective
- On the SQH method for solving optimal control problems with non-smooth state cost functionals or constraints
- Designing rotationally invariant neural networks from PDEs and variational methods
- Nonlinear approximation via compositions
- Universal approximation with quadratic deep networks
- A functional equation with polynomial solutions and application to neural networks
- A global universality of two-layer neural networks with ReLU activations
- Error bounds for approximations with deep ReLU networks
- Neural network with unbounded activation functions is universal approximator
- Optimal stable nonlinear approximation
- Best \(n\)-term approximation of diagonal operators and application to function spaces with mixed smoothness
- Deep vs. shallow networks: an approximation theory perspective
- ReLU networks are universal approximators via piecewise linear or constant functions
- scientific article; zbMATH DE number 683527 (Why is no real title available?)
- Approximation by Combinations of ReLU and Squared ReLU Ridge Functions With <inline-formula> <tex-math notation="LaTeX">$\ell^1$ </tex-math> </inline-formula> and <inline-formula> <tex-math notation="LaTeX">$\ell^0$ </tex-math> </inline-formula> Controls
- Deep Neural Network Approximation Theory
- scientific article; zbMATH DE number 7626778 (Why is no real title available?)
- A deep learning approach to Reduced Order Modelling of parameter dependent partial differential equations
- Theoretical issues in deep networks
- Deep ReLU Networks Overcome the Curse of Dimensionality for Generalized Bandlimited Functions
- Neural parametric Fokker-Planck equation
- Deep neural network surrogates for nonsmooth quantities of interest in shape uncertainty quantification
- A note on the applications of one primary function in deep neural networks
- Deep learning-based approximation of Goldbach partition function
- Comparative studies on mesh-free deep neural network approach versus finite element method for solving coupled nonlinear hyperbolic/wave equations
- PowerNet: efficient representations of polynomials and smooth functions by deep neural networks with rectified power units
- Deep ReLU networks and high-order finite element methods
- Error bounds for approximations with deep ReLU neural networks in \(W^{s , p}\) norms
- Better approximations of high dimensional smooth functions by deep neural networks with rectified power units
- Dying ReLU and initialization: theory and numerical examples
- A note on the expressive power of deep rectified linear unit networks in high-dimensional spaces
- Spline representation and redundancies of one-dimensional ReLU neural network models
- Expressivity of Deep Neural Networks
- Sparse Deep Neural Network for Nonlinear Partial Differential Equations
- Neural network approximation
- Simultaneous neural network approximation for smooth functions
- Approximation capabilities of neural networks on unbounded domains
- Deep ReLU neural network approximation in Bochner spaces and applications to parametric PDEs
- Universality of gradient descent neural network training
- A convergent deep learning algorithm for approximation of polynomials
- Convergence of deep convolutional neural networks
- Mesh-informed neural networks for operator learning in finite element spaces
- ReLU neural networks of polynomial size for exact maximum flow computation
- Approximation error for neural network operators by an averaged modulus of smoothness
- Exponential ReLU neural network approximation rates for point and edge singularities
- Neural ODE Control for Classification, Approximation, and Transport
- Transferable neural networks for partial differential equations
- SignReLU neural network and its approximation ability
- Random neural networks in the infinite width limit as Gaussian processes
- A multivariate Riesz basis of ReLU neural networks
- Collocation approximation by deep neural ReLU networks for parametric and stochastic PDEs with lognormal inputs
- Error bounds for approximations using multichannel deep convolutional neural networks with downsampling
- Connections between numerical algorithms for PDEs and neural networks
- Approximation of compositional functions with ReLU neural networks
- Deep learning via dynamical systems: an approximation perspective
- Estimating a regression function in exponential families by model selection
- Provable Training of a ReLU Gate with an Iterative Non-Gradient Algorithm
- Approximation in shift-invariant spaces with deep ReLU neural networks
- Neural networks with ReLU powers need less depth
- Low dimensional approximation and generalization of multivariate functions on smooth manifolds using deep ReLU neural networks
- Expressive power of ReLU and step networks under floating-point operations
- Deep ReLU networks and high-order finite element methods. II: Chebyšev emulation
- Alternating minimization for regression with tropical rational functions
- Robust nonparametric regression based on deep ReLU neural networks
- On the latent dimension of deep autoencoders for reduced order modeling of PDEs parametrized by random fields
- Improving the expressive power of deep neural networks through integral activation transform
- Approximation results for gradient flow trained shallow neural networks in \(1d\)
- Sampling complexity of deep approximation spaces
- Strong and weak sharp bounds for neural network operators in Sobolev-Orlicz spaces and their quantitative extensions to Orlicz spaces
- Approximation and gradient descent training with neural networks
- Optimal linear B-spline approximation via Kolmogorov superposition theorem and its applications
- Do stable neural networks exist for classification problems? -- A new view on stability in AI
- Efficient shallow Ritz method for 1D diffusion problems
- ALM-PINN: an adaptive physical informed neural network optimized by Levenberg-Marquardt for efficient solution of singular perturbation problems
- Successive affine learning for deep neural networks
- On the optimal approximation of Sobolev and Besov functions using deep ReLU neural networks
- Learning theory of distribution regression with neural networks
- Spectral complexity of deep neural networks
- Approximation results for gradient flow trained neural networks
- Statistical guarantees of group-invariant GANs
- Approximation of functions of several variables by deep operators activated by sigmoidal functions and rectified power units
- Information preservation with Wasserstein autoencoders: generation consistency and adversarial robustness
- A learnable activation function based on orthogonal Legendre wavelets
This page was built for publication: Nonlinear approximation and (deep) ReLU networks
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2117331)
