Optimal Approximation with Sparsely Connected Deep Neural Networks

DOI10.1137/18M118709XzbMath1499.41029arXiv1705.01714MaRDI QIDQ5025773

Gitta Kutyniok, Philipp Petersen, Helmut Bölcskei, Philipp Grohs

Publication date: 3 February 2022

Published in: SIAM Journal on Mathematics of Data Science (Search for Journal in Brave)

Full work available at URL: https://arxiv.org/abs/1705.01714

zbMATH Keywords

waveletsneural networksfunction approximationshearletssparse connectivityoptimal sparse approximation

Mathematics Subject Classification ID

Nontrigonometric harmonic analysis involving wavelets and other special systems (42C40) Artificial neural networks and deep learning (68T07) Signal theory (characterization, reconstruction, filtering, etc.) (94A12) General harmonic expansions, frames (42C15) Rate of convergence, degree of approximation (41A25) Approximation by arbitrary nonlinear expressions; widths and entropy (41A46) Rate-distortion theory in information and communication theory (94A34)

Related Items

Analysis of the Generalization Error: Empirical Risk Minimization over Deep Artificial Neural Networks Overcomes the Curse of Dimensionality in the Numerical Approximation of Black--Scholes Partial Differential Equations ⋮ Non intrusive reduced order modeling of parametrized PDEs by kernel POD and neural networks ⋮ Neural network approximation ⋮ A Proof that Artificial Neural Networks Overcome the Curse of Dimensionality in the Numerical Approximation of Black–Scholes Partial Differential Equations ⋮ Metric entropy limits on recurrent neural network learning of linear dynamical systems ⋮ A neural network multigrid solver for the Navier-Stokes equations ⋮ A deep learning approach to Reduced Order Modelling of parameter dependent partial differential equations ⋮ Quantitative Approximation Results for Complex-Valued Neural Networks ⋮ Approximations with deep neural networks in Sobolev time-space ⋮ Full error analysis for the training of deep neural networks ⋮ Wasserstein generative adversarial uncertainty quantification in physics-informed neural networks ⋮ Theory of deep convolutional neural networks: downsampling ⋮ Bayesian neural network priors for edge-preserving inversion ⋮ Approximation bounds for norm constrained neural networks with applications to regression and GANs ⋮ Simultaneous neural network approximation for smooth functions ⋮ A deep network construction that adapts to intrinsic dimensionality beyond the domain ⋮ Theory of deep convolutional neural networks. III: Approximating radial functions ⋮ Randomized neural network with Petrov-Galerkin methods for solving linear and nonlinear partial differential equations ⋮ Rates of approximation by ReLU shallow neural networks ⋮ An introduction to the mathematics of deep learning ⋮ Three ways to solve partial differential equations with neural networks — A review ⋮ Deep dynamic modeling with just two time points: Can we still allow for individual trajectories? ⋮ Deep learning methods for partial differential equations and related parameter identification problems ⋮ On decision regions of narrow deep neural networks ⋮ On instability mechanisms for inverse problems ⋮ Sparsity in long-time control of neural ODEs ⋮ Overall error analysis for the training of deep neural networks via stochastic gradient descent with random initialisation ⋮ Consistent Sparse Deep Learning: Theory and Computation ⋮ The Lasso with structured design and entropy of (absolute) convex hulls ⋮ Limitations of neural network training due to numerical instability of backpropagation ⋮ Unnamed Item ⋮ SignReLU neural network and its approximation ability ⋮ Invariant spectral foliations with applications to model order reduction and synthesis ⋮ Phase transitions in rate distortion theory and deep learning ⋮ A multivariate Riesz basis of ReLU neural networks ⋮ Lower bounds for artificial neural network approximations: a proof that shallow neural networks fail to overcome the curse of dimensionality ⋮ Neural network approximation and estimation of classifiers with classification boundary in a Barron class ⋮ Stability of iterated dyadic filter banks ⋮ Learning ability of interpolating deep convolutional neural networks ⋮ Deep learning for inverse problems with unknown operator ⋮ The mathematics of artificial intelligence ⋮ Applied harmonic analysis and data science. Abstracts from the workshop held November 28 -- December 4, 2021 (hybrid meeting) ⋮ Approximation with one-bit polynomials in Bernstein form ⋮ Deep learning in high dimension: Neural network expression rates for generalized polynomial chaos expansions in UQ ⋮ Affine symmetries and neural network identifiability ⋮ Nonparametric regression using deep neural networks with ReLU activation function ⋮ Deep ReLU networks and high-order finite element methods ⋮ Superposition, reduction of multivariable problems, and approximation ⋮ Error bounds for approximations with deep ReLU neural networks in Ws,p norms ⋮ Rectified deep neural networks overcome the curse of dimensionality for nonsmooth value functions in zero-sum games of nonlinear stiff systems ⋮ Approximation in shift-invariant spaces with deep ReLU neural networks ⋮ Integral representations of shallow neural network with Rectified Power Unit activation function ⋮ Mini-workshop: Deep learning and inverse problems. Abstracts from the mini-workshop held March 4--10, 2018 ⋮ Deep Network Approximation for Smooth Functions ⋮ Learning on dynamic statistical manifolds ⋮ Butterfly-Net: Optimal Function Representation Based on Convolutional Neural Networks ⋮ Deep Nitsche Method: Deep Ritz Method with Essential Boundary Conditions ⋮ Efficient approximation of solutions of parametric linear transport equations by ReLU DNNs ⋮ Topological properties of the set of functions generated by neural networks of fixed size ⋮ Advances in verification of ReLU neural networks ⋮ Numerical solution of the parametric diffusion equation by deep neural networks ⋮ Universality of deep convolutional neural networks ⋮ Gabor neural networks with proven approximation properties ⋮ Solving the Kolmogorov PDE by means of deep learning ⋮ Equivalence of approximation by convolutional neural networks and fully-connected networks ⋮ Approximation rates for neural networks with encodable weights in smoothness spaces ⋮ Variational Monte Carlo -- bridging concepts of machine learning and high-dimensional partial differential equations ⋮ Theory of deep convolutional neural networks. II: Spherical analysis ⋮ High-dimensional distribution generation through deep neural networks ⋮ Solving inverse problems using data-driven models ⋮ Constructive deep ReLU neural network approximation ⋮ Deep hedging ⋮ Sparsest piecewise-linear regression of one-dimensional data ⋮ Data driven approximation of parametrized PDEs by reduced basis and neural networks ⋮ Deep Network With Approximation Error Being Reciprocal of Width to Power of Square Root of Depth ⋮ BOUNDS ON MULTI-ASSET DERIVATIVES VIA NEURAL NETWORKS ⋮ Structure-preserving deep learning ⋮ Higher-Order Quasi-Monte Carlo Training of Deep Neural Networks ⋮ A phase transition for finding needles in nonlinear haystacks with LASSO artificial neural networks ⋮ Approximation of Smoothness Classes by Deep Rectifier Networks ⋮ Spline representation and redundancies of one-dimensional ReLU neural network models ⋮ Lipschitz Certificates for Layered Network Structures Driven by Averaged Activation Operators ⋮ DNN expression rate analysis of high-dimensional PDEs: application to option pricing ⋮ A theoretical analysis of deep neural networks and parametric PDEs ⋮ Nonlinear approximation and (deep) ReLU networks ⋮ Neural network identifiability for a family of sigmoidal nonlinearities ⋮ Approximation spaces of deep neural networks ⋮ Exponential ReLU DNN expression of holomorphic maps in high dimension

Uses Software

• ImageNet
• AlexNet

Cites Work

• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• AlexNet
• \(\alpha\)-Molecules
• Parabolic molecules
• Cartoon approximation with \(\alpha\)-curvelets
• Compactly supported shearlets are optimally sparse
• Approximation by superposition of sigmoidal and radial basis functions
• Lower bounds for approximation by MLP neural networks
• Approximation and estimation bounds for artificial neural networks
• Unconditional bases are optimal bases for data compression and for statistical estimation
• Foundations of time-frequency analysis
• Multilayer feedforward networks are universal approximators
• Provable approximation properties for deep neural networks
• Approximation properties of a multilayered feedforward artificial neural network
• Degree of approximation by neural and translation networks with a single hidden layer
• Approximation of functions and their derivatives: A neural network implementation with applications
• Optimal approximation of piecewise smooth functions using deep ReLU neural networks
• Error bounds for approximations with deep ReLU networks
• Wave atoms and sparsity of oscillatory patterns
• Ridgelets and the Representation of Mutilated Sobolev Functions
• On the mathematical foundations of learning
• Deep vs. shallow networks: An approximation theory perspective
• Optimally Sparse Data Representations
• Support Vector Machines
• Ten Lectures on Wavelets
• Universal approximation bounds for superpositions of a sigmoidal function
• Neural Networks for Localized Approximation
• New tight frames of curvelets and optimal representations of objects with piecewise C² singularities
• Chaos in Dynamical Systems
• Deep Neural Network Approximation Theory
• Equivalence of approximation by convolutional neural networks and fully-connected networks
• Learning representations by back-propagating errors
• A logical calculus of the ideas immanent in nervous activity
• Approximation by superpositions of a sigmoidal function
• Sparse components of images and optimal atomic decompositions
• Tree approximation and optimal encoding