Nonlinear approximation and (deep) ReLU networks

DOI10.1007/s00365-021-09548-zzbMath1501.41003arXiv1905.02199OpenAlexW3160447895MaRDI QIDQ2117331

Simon Foucart, Ronald A. DeVore, Ingrid Daubechies, B. Hanin, Guergana Petrova

Publication date: 21 March 2022

Published in: Constructive Approximation (Search for Journal in Brave)

Full work available at URL: https://arxiv.org/abs/1905.02199

zbMATH Keywords

neural networksexpressivenessapproximation powerrectified linear unit (ReLU)

Mathematics Subject Classification ID

Artificial neural networks and deep learning (68T07) Neural networks for/in biological studies, artificial life and related topics (92B20) Neural nets applied to problems in time-dependent statistical mechanics (82C32) Rate of convergence, degree of approximation (41A25) Approximation by other special function classes (41A30) Approximation by arbitrary nonlinear expressions; widths and entropy (41A46)

Related Items

Machine learning design of volume of fluid schemes for compressible flows ⋮ Sparse Deep Neural Network for Nonlinear Partial Differential Equations ⋮ Neural network approximation ⋮ Thermodynamically consistent physics-informed neural networks for hyperbolic systems ⋮ A mesh-free method using piecewise deep neural network for elliptic interface problems ⋮ A deep learning approach to Reduced Order Modelling of parameter dependent partial differential equations ⋮ Theoretical issues in deep networks ⋮ On the SQH method for solving optimal control problems with non-smooth state cost functionals or constraints ⋮ Designing rotationally invariant neural networks from PDEs and variational methods ⋮ Neural Parametric Fokker--Planck Equation ⋮ Optimal stable nonlinear approximation ⋮ Deep Neural Network Surrogates for Nonsmooth Quantities of Interest in Shape Uncertainty Quantification ⋮ A note on the applications of one primary function in deep neural networks ⋮ Simultaneous neural network approximation for smooth functions ⋮ Deep ReLU neural network approximation in Bochner spaces and applications to parametric PDEs ⋮ Best \(n\)-term approximation of diagonal operators and application to function spaces with mixed smoothness ⋮ Universality of gradient descent neural network training ⋮ A convergent deep learning algorithm for approximation of polynomials ⋮ Convergence of deep convolutional neural networks ⋮ Mesh-informed neural networks for operator learning in finite element spaces ⋮ Approximation error for neural network operators by an averaged modulus of smoothness ⋮ Exponential ReLU neural network approximation rates for point and edge singularities ⋮ Neural ODE Control for Classification, Approximation, and Transport ⋮ Transferable neural networks for partial differential equations ⋮ Random neural networks in the infinite width limit as Gaussian processes ⋮ A multivariate Riesz basis of ReLU neural networks ⋮ Collocation approximation by deep neural ReLU networks for parametric and stochastic PDEs with lognormal inputs ⋮ Error bounds for approximations using multichannel deep convolutional neural networks with downsampling ⋮ Connections between numerical algorithms for PDEs and neural networks ⋮ Deep learning via dynamical systems: an approximation perspective ⋮ Estimating a regression function in exponential families by model selection ⋮ A functional equation with polynomial solutions and application to neural networks ⋮ Unnamed Item ⋮ Unnamed Item ⋮ High-dimensional distribution generation through deep neural networks ⋮ The construction and approximation of ReLU neural network operators ⋮ Stable recovery of entangled weights: towards robust identification of deep neural networks from minimal samples ⋮ Information theory and recovery algorithms for data fusion in Earth observation ⋮ Spline representation and redundancies of one-dimensional ReLU neural network models

Uses Software

• ImageNet
• GNMT
• AlexNet

Cites Work

• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• The Takagi function: a survey
• Weierstrass' function and chaos
• Multilayer feedforward networks are universal approximators
• Provable approximation properties for deep neural networks
• Optimal nonlinear approximation
• Wavelet compression and nonlinear \(n\)-widths
• Exponential convergence of the deep neural network approximation for analytic functions
• Error bounds for approximations with deep ReLU networks
• Deep vs. shallow networks: An approximation theory perspective
• Neural Networks for Localized Approximation
• Deep learning in high dimension: Neural network expression rates for generalized polynomial chaos expansions in UQ
• Optimal Approximation with Sparsely Connected Deep Neural Networks
• Deep Network Approximation for Smooth Functions
• Deep Network Approximation Characterized by Number of Neurons
• Approximation by superpositions of a sigmoidal function