Analysis of the rate of convergence of fully connected deep neural network regression estimates with smooth activation function

DOI10.1016/J.JMVA.2020.104695MaRDI QIDQ2222227zbMATH OpenOpenAlexFDO

Authors Sophie Langer

Publication date 26 January 2021

Published in Journal of Multivariate Analysis (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2010.06168

zbMATH Keywords

curse of dimensionalitydeep learningneural networksnonparametric regressionrate of convergence

Mathematics Subject Classification ID

Rate of convergence, degree of approximation (41A25) Nonparametric regression and quantile regression (62G08) Neural nets and related approaches to inference from stochastic processes (62M45) Artificial neural networks and deep learning (68T07) Neural nets applied to problems in time-dependent statistical mechanics (82C32)

Abstract: This article contributes to the current statistical theory of deep neural networks (DNNs). It was shown that DNNs are able to circumvent the so--called curse of dimensionality in case that suitable restrictions on the structure of the regression function hold. In most of those results the tuning parameter is the sparsity of the network, which describes the number of non-zero weights in the network. This constraint seemed to be the key factor for the good rate of convergence results. Recently, the assumption was disproved. In particular, it was shown that simple fully connected DNNs can achieve the same rate of convergence. Those fully connected DNNs are based on the unbounded ReLU activation function. In this article we extend the results to smooth activation functions, i.e., to the sigmoid activation function. It is shown that estimators based on fully connected DNNs with sigmoid activation function also achieve the minimax rates of convergence (up to $lnn$-factors). In our result the number of hidden layers is fixed, the number of neurons per layer tends to infinity for sample size tending to infinity and a bound for the weights in the network is given.

Review text

There is an investigation on regression estimators based on deep neural networks (DNN). In a previous article, [the author and \textit{M. Kohler}, ``On the rate of convergence of fully connected deep neural network regression estimates, Preprint, \url{arXiv:1908.11133}], neural networks with rectified linear unit (ReLU) activation function have been considered. The question here is, if the same rate of convergence for fully connected deep neural networks regression estimators with smooth activation function -- the sigmoid -- can be achieved. Indeed, the main result of the present paper, proves that under a set of sufficient conditions, the \(L_2\) -errors of least squares neural network regression estimators based on a set of fully connected DNNs with a fixed number of layers, achieve a similar rate of convergence as in the mentioned article.

Recommendations

• On the rate of convergence of fully connected deep neural network regression estimates
• Approximating smooth functions by deep neural networks with sigmoid activation function
• Estimation of a regression function on a manifold by fully connected deep neural networks
• Nonparametric regression using deep neural networks with ReLU activation function
• scientific article; zbMATH DE number 6178473

Cites work

• A distribution-free theory of nonparametric regression
• Adaptive regression estimation with multilayer feedforward neural networks
• Additive regression and other nonparametric models
• Approximation and estimation bounds for artificial neural networks
• Distribution-free consistency results in nonparametric discrimination and regression function estimation
• Estimation of a Regression Function by Maxima of Minima of Linear Functions
• scientific article; zbMATH DE number 17222 (Why is no real title available?)
• scientific article; zbMATH DE number 51537 (Why is no real title available?)
• scientific article; zbMATH DE number 893887 (Why is no real title available?)
• Investigating Smooth Multiple Regression by the Method of Average Derivatives
• Nonparametric Regression Based on Hierarchical Interaction Models
• Nonparametric regression using deep neural networks with ReLU activation function
• On deep learning as a remedy for the curse of dimensionality in nonparametric regression
• Optimal global rates of convergence for nonparametric regression
• Optimal smoothing in single-index models
• Penalized Spline Estimation for Partially Linear Single-Index Models
• Rate-optimal estimation for a general class of nonparametric regression models with unknown link functions
• The use of polynomial splines and their tensor products in multivariate function estimation. (With discussion)
• Universal approximation bounds for superpositions of a sigmoidal function
• Variable selection for the single-index model

Cited in

(11)

• Estimation error analysis of deep learning on the regression problem on the variable exponent Besov space
• On the rate of convergence of fully connected deep neural network regression estimates
• Nonparametric regression using deep neural networks with ReLU activation function
• On deep learning as a remedy for the curse of dimensionality in nonparametric regression
• Estimation of a regression function on a manifold by fully connected deep neural networks
• On the effect of the activation function on the distribution of hidden nodes in a deep network
• Weak and strong convergence analysis of Elman neural networks via weight decay regularization
• Fast convergence rates of deep neural networks for classification
• Learning sparse and smooth functions by deep sigmoid nets
• Convergence rates for shallow neural networks learned by gradient descent
• A Quantitative Functional Central Limit Theorem for Shallow Neural Networks

This page was built for publication: Analysis of the rate of convergence of fully connected deep neural network regression estimates with smooth activation function

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2222227)