Deep vs. shallow networks: an approximation theory perspective
From MaRDI portal
Abstract: The paper briefy reviews several recent results on hierarchical architectures for learning from examples, that may formally explain the conditions under which Deep Convolutional Neural Networks perform much better in function approximation problems than shallow, one-hidden layer architectures. The paper announces new results for a non-smooth activation function - the ReLU function - used in present-day neural networks, as well as for the Gaussian networks. We propose a new definition of relative dimension to encapsulate different notions of sparsity of a function class that can possibly be exploited by deep networks but not by shallow ones to drastically reduce the complexity required for approximation and learning.
Recommendations
Cites work
- scientific article; zbMATH DE number 52869 (Why is no real title available?)
- scientific article; zbMATH DE number 1231230 (Why is no real title available?)
- scientific article; zbMATH DE number 1972818 (Why is no real title available?)
- scientific article; zbMATH DE number 2115052 (Why is no real title available?)
- scientific article; zbMATH DE number 2204742 (Why is no real title available?)
- scientific article; zbMATH DE number 3329342 (Why is no real title available?)
- Limitations of the approximation capabilities of neural networks with one hidden layer
- Neocognition: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position
- Weighted quadrature formulas and approximation by zonal function networks on the sphere
- When is approximation by Gaussian networks necessarily a linear process?
Cited in
(only showing first 100 items - show all)- Why does deep and cheap learning work so well?
- Saturation classes for MAX-product neural network operators activated by sigmoidal functions
- Applied harmonic analysis and data processing. Abstracts from the workshop held March 25--31, 2018
- Function approximation with zonal function networks with activation functions analogous to the rectified linear unit functions
- Estimates for the neural network operators of the max-product type with continuous and \(p\)-integrable functions
- Asymptotic expansion for neural network operators of the Kantorovich type and high order of approximation
- Deep autoencoder based energy method for the bending, vibration, and buckling analysis of Kirchhoff plates with transfer learning
- On the rate of convergence of fully connected deep neural network regression estimates
- A direct approach for function approximation on data defined manifolds
- An analytic layer-wise deep learning framework with applications to robotics
- Optimal adaptive control of partially uncertain linear continuous-time systems with state delay
- Why rectified linear activation functions? Why max-pooling? A possible explanation
- Stable recovery of entangled weights: towards robust identification of deep neural networks from minimal samples
- A measure theoretical approach to the mean-field maximum principle for training NeurODEs
- Error bounds for ReLU networks with depth and width parameters
- Nonlinear approximation and (deep) ReLU networks
- Approximation spaces of deep neural networks
- On sharpness of an error bound for deep ReLU network approximation
- Approximation properties of deep ReLU CNNs
- Uniform approximation rates and metric entropy of shallow neural networks
- Voronovskaja type theorems and high-order convergence neural network operators with sigmoidal functions
- On the approximation by single hidden layer feedforward neural networks with fixed weights
- Probabilistic lower bounds for approximation by shallow perceptron networks
- An analysis of training and generalization errors in shallow and deep networks
- Universal approximation with quadratic deep networks
- Function approximation by deep networks
- Topology optimization based on deep representation learning (DRL) for compliance and stress-constrained design
- Nonparametric regression using deep neural networks with ReLU activation function
- Convergence of the deep BSDE method for coupled FBSDEs
- Limitations of shallow nets approximation
- A linear relation between input and first layer in neural networks
- Universality of deep convolutional neural networks
- On deep learning as a remedy for the curse of dimensionality in nonparametric regression
- Application of deep learning neural network to identify collision load conditions based on permanent plastic deformation of shell structures
- Super-resolution meets machine learning: approximation of measures
- Estimation of a regression function on a manifold by fully connected deep neural networks
- Local approximation of operators
- Mini-workshop: Analysis of data-driven optimal control. Abstracts from the mini-workshop held May 9--15, 2021 (hybrid meeting)
- scientific article; zbMATH DE number 6500551 (Why is no real title available?)
- Deep distributed convolutional neural networks: universality
- Deep learning in high dimension: neural network expression rates for generalized polynomial chaos expansions in UQ
- Generalization error of minimum weighted norm and kernel interpolation
- The gap between theory and practice in function approximation with deep neural networks
- Deep Neural Network Approximation Theory
- Optimal approximation with sparsely connected deep neural networks
- On some similarities and differences between deep neural networks and kernel learning machines
- scientific article; zbMATH DE number 7626756 (Why is no real title available?)
- High-Dimensional Learning Under Approximate Sparsity with Applications to Nonsmooth Estimation and Regularized Neural Networks
- Feedforward neural networks and compositional functions with applications to dynamical systems
- The Principles of Deep Learning Theory
- What Kinds of Functions Do Deep Neural Networks Learn? Insights from Variational Spline Theory
- Theoretical issues in deep networks
- Full error analysis for the training of deep neural networks
- Rectified deep neural networks overcome the curse of dimensionality for nonsmooth value functions in zero-sum games of nonlinear stiff systems
- Topology of deep neural networks
- Optimization under uncertainty explains empirical success of deep learning heuristics
- Butterfly-net: optimal function representation based on convolutional neural networks
- Quantitative estimates involving K-functionals for neural network-type operators
- Deep neural networks for rotation-invariance approximation and learning
- Robust randomized optimization with k nearest neighbors
- Approximating functions with multi-features by deep convolutional neural networks
- Low-rank approximation of continuous functions in Sobolev spaces with dominating mixed smoothness
- A Proof that Artificial Neural Networks Overcome the Curse of Dimensionality in the Numerical Approximation of Black–Scholes Partial Differential Equations
- Stable parameterization of continuous and piecewise-linear functions
- A deep network construction that adapts to intrinsic dimensionality beyond the domain
- Theory of deep convolutional neural networks. III: Approximating radial functions
- De Rham compatible deep neural network FEM
- Approximating smooth and sparse functions by deep neural networks: optimal approximation rates and saturation
- Rates of approximation by ReLU shallow neural networks
- Three ways to solve partial differential equations with neural networks — A review
- Analyzing Decision Polygons of DNN-based Classification Methods
- Kähler geometry of framed quiver moduli and machine learning
- On minimal representations of shallow ReLU networks
- The Kolmogorov-Arnold representation theorem revisited
- Fast convergence rates of deep neural networks for classification
- On decision regions of narrow deep neural networks
- Approximation Analysis of Convolutional Neural Networks
- Time discretization in the solution of parabolic PDEs with ANNs
- MorphoActivation: generalizing ReLU activation function by mathematical morphology
- Overall error analysis for the training of deep neural networks via stochastic gradient descent with random initialisation
- Learning sparse and smooth functions by deep sigmoid nets
- Dunkl analouge of Szász Schurer Beta bivariate operators
- Error convergence and engineering-guided hyperparameter search of PINNs: towards optimized I-FENN performance
- Data augmentation for Bayesian deep learning
- On the number of regions of piecewise linear neural networks
- Lower bounds for artificial neural network approximations: a proof that shallow neural networks fail to overcome the curse of dimensionality
- Approximation of compositional functions with ReLU neural networks
- Convergence rate of DeepONets for learning operators arising from advection-diffusion equations
- Approximation of classifiers by deep perceptron networks
- Neural networks with ReLU powers need less depth
- Statistical insights into deep neural network learning in subspace classification
- Investigating deep energy method applications in thermoelasticity
- Compositional sparsity of learnable functions
- Relevant sampling in a reproducing kernel subspace of Orlicz space
- Variational temporal convolutional networks for I-FENN thermoelasticity
- Approximation rates for deep calibration of (rough) stochastic volatility models
- Learning and approximating piecewise smooth functions by deep sigmoid neural networks
- Choice of interior penalty coefficient for interior penalty discontinuous Galerkin method for Biot's system by employing machine learning
- Approximating functions with additive ridge features by deep \(\mathrm{ReLU}^k\) convolutional neural networks
- Approximation and estimation capability of vision transformers for hierarchical compositional models
This page was built for publication: Deep vs. shallow networks: an approximation theory perspective
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2835988)
