On the rate of convergence of fully connected deep neural network regression estimates

DOI10.1214/20-AOS2034zbMATH Open1486.62112arXiv1908.11133MaRDI QIDQ2054491FDOQ2054491

Authors: Michael Kohler, Sophie Langer 

Publication date: 3 December 2021

Published in: The Annals of Statistics (Search for Journal in Brave)

Abstract: Recent results in nonparametric regression show that deep learning, i.e., neural network estimates with many hidden layers, are able to circumvent the so-called curse of dimensionality in case that suitable restrictions on the structure of the regression function hold. One key feature of the neural networks used in these results is that their network architecture has a further constraint, namely the network sparsity. In this paper we show that we can get similar results also for least squares estimates based on simple fully connected neural networks with ReLU activation functions. Here either the number of neurons per hidden layer is fixed and the number of hidden layers tends to infinity suitably fast for sample size tending to infinity, or the number of hidden layers is bounded by some logarithmic factor in the sample size and the number of neurons per hidden layer tends to infinity suitably fast for sample size tending to infinity. The proof is based on new approximation results concerning deep neural networks.

Full work available at URL: https://arxiv.org/abs/1908.11133

Recommendations

• Analysis of the rate of convergence of fully connected deep neural network regression estimates with smooth activation function
• Estimation of a regression function on a manifold by fully connected deep neural networks
• On deep learning as a remedy for the curse of dimensionality in nonparametric regression
• Nonparametric regression using deep neural networks with ReLU activation function
• Function approximation by deep neural networks with parameters \(\{0, \pm \frac{1}{2}, \pm 1,2\}\)

zbMATH Keywords

deep learningnonparametric regressionrate of convergencecurse of dimensionalityneural networks

Mathematics Subject Classification ID

Nonparametric regression and quantile regression (62G08) Asymptotic properties of nonparametric inference (62G20) Artificial neural networks and deep learning (68T07)

Cites Work

• Optimal smoothing in single-index models
• Universal approximation bounds for superpositions of a sigmoidal function
• Penalized Spline Estimation for Partially Linear Single-Index Models
• Additive regression and other nonparametric models
• Investigating Smooth Multiple Regression by the Method of Average Derivatives
• Variable selection for the single-index model
• Optimal global rates of convergence for nonparametric regression
• The use of polynomial splines and their tensor products in multivariate function estimation. (With discussion)
• Title not available (Why is that?)
• A distribution-free theory of nonparametric regression
• Rate-optimal estimation for a general class of nonparametric regression models with unknown link functions
• Distribution-free consistency results in nonparametric discrimination and regression function estimation
• Title not available (Why is that?)
• Approximation and estimation bounds for artificial neural networks
• Adaptive regression estimation with multilayer feedforward neural networks
• On deep learning as a remedy for the curse of dimensionality in nonparametric regression
• Deep vs. shallow networks: an approximation theory perspective
• Nonparametric Regression Based on Hierarchical Interaction Models
• Nonparametric regression using deep neural networks with ReLU activation function
• Deep Neural Network Approximation Theory
• Deep Network Approximation for Smooth Functions

Cited In (33)

• Calibrating multi-dimensional complex ODE from noisy data via deep neural networks
• Convergence rates for shallow neural networks learned by gradient descent
• Asymptotic properties of one-layer artificial neural networks with sparse connectivity
• Local convergence rates of the nonparametric least squares estimator with applications to transfer learning
• Analysis of convolutional neural network image classifiers in a hierarchical max-pooling model with additional local pooling
• Convergence rates of deep ReLU networks for multiclass classification
• A Deep Generative Approach to Conditional Sampling
• Deep learning based on randomized quasi-Monte Carlo method for solving linear Kolmogorov partial differential equation
• Recovering the source term in elliptic equation via deep learning: method and convergence analysis
• Estimation of a regression function on a manifold by fully connected deep neural networks
• An error analysis for deep binary classification with sigmoid loss
• Robust nonparametric regression based on deep ReLU neural networks
• Deep nonparametric regression on approximate manifolds: nonasymptotic error bounds with polynomial prefactors
• Convergence analysis for over-parameterized deep learning
• Mini-workshop: Nonlinear approximation of high-dimensional functions in scientific computing. Abstracts from the mini-workshop held October 15--20, 2023
• Layer sparsity in neural networks
• Statistical theory for image classification using deep convolutional neural network with cross-entropy loss under the hierarchical max-pooling model
• How do noise tails impact on deep ReLU networks?
• Factor Augmented Sparse Throughput Deep ReLU Neural Networks for High Dimensional Regression
• Adaptive variational Bayes: optimality, computation and applications
• Asset pricing with neural networks: significance tests
• Adaptive deep learning for nonlinear time series models
• Optimal convergence rates of deep neural networks in a classification setting
• Analysis of the rate of convergence of two regression estimates defined by neural features which are easy to implement
• Intrinsic and extrinsic deep learning on manifolds
• Fast convergence rates of deep neural networks for classification
• Sampling complexity of deep approximation spaces
• Error analysis for empirical risk minimization over clipped ReLU networks in solving linear Kolmogorov partial differential equations
• On the rate of convergence of a deep recurrent neural network estimate in a regression problem with dependent data
• Applied harmonic analysis and data science. Abstracts from the workshop held April 21--26, 2024
• Asymptotic properties of neural network sieve estimators
• On the rate of convergence of image classifiers based on convolutional neural networks
• A deep learning method for pricing high-dimensional American-style options via state-space partition

This page was built for publication: On the rate of convergence of fully connected deep neural network regression estimates

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2054491)