Deep vs. shallow networks: An approximation theory perspective

DOI10.1142/S0219530516400042zbMath1355.68233arXiv1608.03287OpenAlexW2513671774WikidataQ125020678 ScholiaQ125020678MaRDI QIDQ2835988

Tomaso Poggio, Hrushikesh N. Mhaskar

Publication date: 30 November 2016

Published in: Analysis and Applications (Search for Journal in Brave)

Full work available at URL: https://arxiv.org/abs/1608.03287

zbMATH Keywords

Gaussian networksshallow networksdeep networksReLU networksblessed representation

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05) Rate of convergence, degree of approximation (41A25) Approximation by arbitrary nonlinear expressions; widths and entropy (41A46)

Related Items

Low-rank approximation of continuous functions in Sobolev spaces with dominating mixed smoothness ⋮ A Proof that Artificial Neural Networks Overcome the Curse of Dimensionality in the Numerical Approximation of Black–Scholes Partial Differential Equations ⋮ On sharpness of an error bound for deep ReLU network approximation ⋮ High-Dimensional Learning Under Approximate Sparsity with Applications to Nonsmooth Estimation and Regularized Neural Networks ⋮ Feedforward Neural Networks and Compositional Functions with Applications to Dynamical Systems ⋮ Deep distributed convolutional neural networks: Universality ⋮ Theoretical issues in deep networks ⋮ Full error analysis for the training of deep neural networks ⋮ Why does deep and cheap learning work so well? ⋮ Voronovskaja type theorems and high-order convergence neural network operators with sigmoidal functions ⋮ On the approximation by single hidden layer feedforward neural networks with fixed weights ⋮ Saturation classes for MAX-product neural network operators activated by sigmoidal functions ⋮ An analysis of training and generalization errors in shallow and deep networks ⋮ Universal approximation with quadratic deep networks ⋮ Estimation of a regression function on a manifold by fully connected deep neural networks ⋮ Stable parameterization of continuous and piecewise-linear functions ⋮ A deep network construction that adapts to intrinsic dimensionality beyond the domain ⋮ Theory of deep convolutional neural networks. III: Approximating radial functions ⋮ De Rham compatible deep neural network FEM ⋮ Approximating smooth and sparse functions by deep neural networks: optimal approximation rates and saturation ⋮ Rates of approximation by ReLU shallow neural networks ⋮ Three ways to solve partial differential equations with neural networks — A review ⋮ Kähler geometry of framed quiver moduli and machine learning ⋮ The Kolmogorov-Arnold representation theorem revisited ⋮ On decision regions of narrow deep neural networks ⋮ Approximation Analysis of Convolutional Neural Networks ⋮ Time discretization in the solution of parabolic PDEs with ANNs ⋮ Function approximation by deep networks ⋮ Overall error analysis for the training of deep neural networks via stochastic gradient descent with random initialisation ⋮ Learning sparse and smooth functions by deep sigmoid nets ⋮ Dunkl analouge of Szász Schurer Beta bivariate operators ⋮ Error convergence and engineering-guided hyperparameter search of PINNs: towards optimized I-FENN performance ⋮ Data augmentation for Bayesian deep learning ⋮ On the number of regions of piecewise linear neural networks ⋮ Lower bounds for artificial neural network approximations: a proof that shallow neural networks fail to overcome the curse of dimensionality ⋮ Approximation of compositional functions with ReLU neural networks ⋮ Local approximation of operators ⋮ Mini-workshop: Analysis of data-driven optimal control. Abstracts from the mini-workshop held May 9--15, 2021 (hybrid meeting) ⋮ Topology optimization based on deep representation learning (DRL) for compliance and stress-constrained design ⋮ Deep learning in high dimension: Neural network expression rates for generalized polynomial chaos expansions in UQ ⋮ Nonparametric regression using deep neural networks with ReLU activation function ⋮ Rectified deep neural networks overcome the curse of dimensionality for nonsmooth value functions in zero-sum games of nonlinear stiff systems ⋮ Convergence rate of DeepONets for learning operators arising from advection-diffusion equations ⋮ Convergence of the deep BSDE method for coupled FBSDEs ⋮ Applied harmonic analysis and data processing. Abstracts from the workshop held March 25--31, 2018 ⋮ Function approximation with zonal function networks with activation functions analogous to the rectified linear unit functions ⋮ Estimates for the neural network operators of the max-product type with continuous and \(p\)-integrable functions ⋮ Unnamed Item ⋮ Butterfly-Net: Optimal Function Representation Based on Convolutional Neural Networks ⋮ Asymptotic expansion for neural network operators of the Kantorovich type and high order of approximation ⋮ Quantitative estimates involving K-functionals for neural network-type operators ⋮ Deep autoencoder based energy method for the bending, vibration, and buckling analysis of Kirchhoff plates with transfer learning ⋮ A linear relation between input and first layer in neural networks ⋮ Universality of deep convolutional neural networks ⋮ On the rate of convergence of fully connected deep neural network regression estimates ⋮ A direct approach for function approximation on data defined manifolds ⋮ An analytic layer-wise deep learning framework with applications to robotics ⋮ On deep learning as a remedy for the curse of dimensionality in nonparametric regression ⋮ Application of deep learning neural network to identify collision load conditions based on permanent plastic deformation of shell structures ⋮ Deep neural networks for rotation-invariance approximation and learning ⋮ Robust randomized optimization with k nearest neighbors ⋮ Generalization Error of Minimum Weighted Norm and Kernel Interpolation ⋮ Optimal adaptive control of partially uncertain linear continuous-time systems with state delay ⋮ Super-resolution meets machine learning: approximation of measures ⋮ Stable recovery of entangled weights: towards robust identification of deep neural networks from minimal samples ⋮ A measure theoretical approach to the mean-field maximum principle for training NeurODEs ⋮ Error bounds for ReLU networks with depth and width parameters ⋮ Optimal Approximation with Sparsely Connected Deep Neural Networks ⋮ Approximating functions with multi-features by deep convolutional neural networks ⋮ Nonlinear approximation and (deep) ReLU networks ⋮ Approximation spaces of deep neural networks

Cites Work

• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Neocognition: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position
• When is approximation by Gaussian networks necessarily a linear process?
• Limitations of the approximation capabilities of neural networks with one hidden layer
• Weighted quadrature formulas and approximation by zonal function networks on the sphere