Nonparametric regression using deep neural networks with ReLU activation function
From MaRDI portal
Publication:2215715
additive modelsminimax estimation riskmultilayer neural networksnonparametric regressionrectified linear activation functionrectified linear unit (ReLU)wavelets
Nontrigonometric harmonic analysis involving wavelets and other special systems (42C40) Minimax procedures in statistical decision theory (62C20) Nonparametric regression and quantile regression (62G08) Neural nets and related approaches to inference from stochastic processes (62M45) Artificial neural networks and deep learning (68T07)
Abstract: Consider the multivariate nonparametric regression model. It is shown that estimators based on sparsely connected deep neural networks with ReLU activation function and properly chosen network architecture achieve the minimax rates of convergence (up to -factors) under a general composition assumption on the regression function. The framework includes many well-studied structural constraints such as (generalized) additive models. While there is a lot of flexibility in the network architecture, the tuning parameter is the sparsity of the network. Specifically, we consider large networks with number of potential network parameters exceeding the sample size. The analysis gives some insights into why multilayer feedforward neural networks perform well in practice. Interestingly, for ReLU activation function the depth (number of layers) of the neural network architectures plays an important role and our theory suggests that for nonparametric regression, scaling the network depth with the sample size is natural. It is also shown that under the composition assumption wavelet estimators can only achieve suboptimal rates.
The author considers the already classical nonparametric regression model $Y_i =f_0(\mathbf{X} _i) +\epsilon _i $, where $\mathbf{X}_i\in[0,1]^d$, $Y_i\in \mathbb R$, $i=1,\dots,n$, $\mathbf{X} _i$ are the observed vectors, $Y_i$ are the responses and $\epsilon _i$ are the noise variables. The statistical problem is to recover $f_0$ from the samples ($\mathbf{X}_i$, $Y_i$). In the present paper, the estimation of the regression function is done by means of neural networks technology. In the second section, multilayer feedforward artificial neural networks are introduced and concepts as deep neural networks, deep learning, sparsity of the neural network, rectified linear unit (ReLU) are explained. The third section is devoted to the presentation of main results. The regression function is assumed to be a composition of several functions. The author proves that the estimators based on sparsely connected deep neural networks with ReLU activation function and for an appropriate network architecture can achieve minimax rates of convergence. Special cases of compositional constrains on the regression function are discussed in the fourth section. Suboptimality of wavelet series estimators is discussed in the fifth section and a summary on some achievements on statistical theory and neural networks can be found in the sixth section. The seventh section contains the proofs of results. The author reports that additional proofs are to be found in a supplementary material: ``Nonparametric regression using deep neural networks with ReLU activation function, \url{doi:10.1214/19-AOS1875SUPP}.
Recommendations
- On the rate of convergence of fully connected deep neural network regression estimates
- Analysis of the rate of convergence of fully connected deep neural network regression estimates with smooth activation function
- Estimation of a regression function on a manifold by fully connected deep neural networks
- Function approximation by deep neural networks with parameters \(\{0, \pm \frac{1}{2}, \pm 1,2\}\)
- On deep learning as a remedy for the curse of dimensionality in nonparametric regression
Cites work
- A distribution-free theory of nonparametric regression
- A regularity class for the roots of nonnegative functions
- Adaptive regression estimation with multilayer feedforward neural networks
- All of Nonparametric Statistics
- Approximation and estimation bounds for artificial neural networks
- Approximation by superpositions of a sigmoidal function
- Approximation properties of a multilayered feedforward artificial neural network
- Breaking the curse of dimensionality with convex neural networks
- Concentration inequalities and asymptotic results for ratio type empirical processes
- Concentration inequalities and model selection. Ecole d'Eté de Probabilités de Saint-Flour XXXIII -- 2003.
- Deep learning
- Deep vs. shallow networks: an approximation theory perspective
- Error bounds for approximations with deep ReLU networks
- Estimating composite functions by model selection
- scientific article; zbMATH DE number 6378127 (Why is no real title available?)
- scientific article; zbMATH DE number 1981771 (Why is no real title available?)
- scientific article; zbMATH DE number 1405266 (Why is no real title available?)
- Introduction to nonparametric estimation
- Local Rademacher complexities and oracle inequalities in risk minimization. (2004 IMS Medallion Lecture). (With discussions and rejoinder)
- Multilayer feedforward networks are universal approximators
- Neural Network Learning
- Nonasymptotic bounds on the \(L_{2}\) error of neural network regression estimates
- Nonparametric estimation of composite functions
- Nonparametric Regression Based on Hierarchical Interaction Models
- On deep learning as a remedy for the curse of dimensionality in nonparametric regression
- Optimal approximation with sparsely connected deep neural networks
- Rate-optimal estimation for a general class of nonparametric regression models with unknown link functions
- Universal approximation bounds for superpositions of a sigmoidal function
- Universal Approximation Depth and Errors of Narrow Belief Networks with Discrete Units
- Wavelets on the interval and fast wavelet transforms
Cited in
(only showing first 100 items - show all)- Over-parametrized deep neural networks minimizing the empirical risk do not generalize well
- A selective overview of deep learning
- Estimation of an improved surrogate model in uncertainty quantification by neural networks
- Estimation error analysis of deep learning on the regression problem on the variable exponent Besov space
- On the rate of convergence of fully connected deep neural network regression estimates
- Fast generalization error bound of deep learning without scale invariance of activation functions
- Approximation rates for neural networks with encodable weights in smoothness spaces
- Function approximation by deep neural networks with parameters \(\{0, \pm \frac{1}{2}, \pm 1,2\}\)
- Nonparametric regression with modified ReLU networks
- On the rate of convergence of image classifiers based on convolutional neural networks
- Measurement error models: from nonparametric methods to deep neural networks
- High-performance statistical computing in the computing environments of the 2020s
- Robust deep neural network estimation for multi-dimensional functional data
- A new deep neural network algorithm for multiple stopping with applications in options pricing
- Error bounds for ReLU networks with depth and width parameters
- Analysis of convolutional neural network image classifiers in a hierarchical max-pooling model with additional local pooling
- DNN expression rate analysis of high-dimensional PDEs: application to option pricing
- Approximation spaces of deep neural networks
- Mathematical foundations of machine learning. Abstracts from the workshop held March 21--27, 2021 (hybrid meeting)
- Convergence rates of deep ReLU networks for multiclass classification
- Deep learning for the partially linear Cox model
- Discussion to: ``Bayesian graphical models for modern biological applications by Y. Ni, V. Baladandayuthapani, M. Vannucci and F. C. Stingo
- On the SQH method for solving optimal control problems with non-smooth state cost functionals or constraints
- On the minimax optimality and superiority of deep neural network learning over sparse parameter spaces
- Analysis of the rate of convergence of fully connected deep neural network regression estimates with smooth activation function
- Approximating smooth functions by deep neural networks with sigmoid activation function
- Optimal nonparametric inference via deep neural network
- Learning sparse deep neural networks with a spike-and-slab prior
- On deep learning as a remedy for the curse of dimensionality in nonparametric regression
- Estimation of a regression function on a manifold by fully connected deep neural networks
- Smoothing and adaptation of shifted Pólya tree ensembles
- Space-time error estimates for deep neural network approximations for differential equations
- Individualized risk assessment of preoperative opioid use by interpretable neural network regression
- Deep learning for inverse problems with unknown operator
- On the rate of convergence of a deep recurrent neural network estimate in a regression problem with dependent data
- Transport analysis of infinitely deep neural network
- Adaptive estimation of nonparametric functionals
- scientific article; zbMATH DE number 7387620 (Why is no real title available?)
- scientific article; zbMATH DE number 7387621 (Why is no real title available?)
- Deep neural networks for estimation and inference
- Multilinear compressive sensing and an application to convolutional linear networks
- Particle dual averaging: optimization of mean field neural network with global convergence rate analysis*
- Regression Neural Networks with a Highly Robust Loss Function
- Adaptive approximation and generalization of deep neural network with intrinsic dimensionality
- Deep network approximation characterized by number of neurons
- A representer theorem for deep neural networks
- Computer model calibration with time series data using deep learning and quantile regression
- Approximating functions with multi-features by deep convolutional neural networks
- Nonparametric regression on low-dimensional manifolds using deep ReLU networks: function approximation and statistical recovery
- scientific article; zbMATH DE number 7660136 (Why is no real title available?)
- Stationary Density Estimation of Itô Diffusions Using Deep Learning
- Neural network approximation
- Approximation bounds for norm constrained neural networks with applications to regression and GANs
- Conditional predictive inference for stable algorithms
- Neural network approximation: three hidden layers are enough
- A deep network construction that adapts to intrinsic dimensionality beyond the domain
- Theory of deep convolutional neural networks. III: Approximating radial functions
- Inference on heterogeneous treatment effects in high‐dimensional dynamic panels under weak dependence
- Comprehensive study of variational Bayes classification for dense deep neural networks
- The Kolmogorov-Arnold representation theorem revisited
- Fast convergence rates of deep neural networks for classification
- Statistical guarantees for regularized neural networks
- Asymptotic properties of neural network sieve estimators
- Estimation of projection pursuit regression via alternating linearization
- Debiased machine learning of set-identified linear models
- Consistent Sparse Deep Learning: Theory and Computation
- Data augmentation for Bayesian deep learning
- Drift estimation for a multi-dimensional diffusion process using deep neural networks
- Deep Neural Networks with ReLU-Sine-Exponential Activations Break Curse of Dimensionality in Approximation on Hölder Class
- Convergence rates for shallow neural networks learned by gradient descent
- Deep neural network classifier for multidimensional functional data
- A multivariate Riesz basis of ReLU neural networks
- Nonexact oracle inequalities, r-learnability, and fast rates
- Improved Analysis of PINNs: Alleviate the CoD for Compositional Solutions
- Off-Policy Confidence Interval Estimation with Confounded Markov Decision Process
- Optimal deep neural networks by maximization of the approximation power
- Deep learning theory of distribution regression with CNNs
- Deep nonparametric regression on approximate manifolds: nonasymptotic error bounds with polynomial prefactors
- Optimal convergence rates of deep neural networks in a classification setting
- Learning ability of interpolating deep convolutional neural networks
- Adaptive variational Bayes: optimality, computation and applications
- Correction to: ``Nonparametric regression using deep neural networks with ReLU activation function
- Causal inference of general treatment effects using neural networks with a diverging number of confounders
- Asset pricing with neural networks: significance tests
- Assumption-lean falsification tests of rate double-robustness of double-machine-learning estimators
- Analysis of the rate of convergence of two regression estimates defined by neural features which are easy to implement
- Intrinsic and extrinsic deep learning on manifolds
- Multiclass classification for multidimensional functional data through deep neural networks
- Estimating a regression function in exponential families by model selection
- Error assessment of an adaptive finite elements -- neural networks method for an elliptic parametric PDE
- Integral representations of shallow neural network with Rectified Power Unit activation function
- Simultaneous approximation of a smooth function and its derivatives by deep neural networks with piecewise-polynomial activations
- On the approximability and curse of dimensionality of certain classes of high-dimensional functions
- Deep signature algorithm for multidimensional path-dependent options
- Low dimensional approximation and generalization of multivariate functions on smooth manifolds using deep ReLU neural networks
- Estimation of the mean function of functional data via deep neural networks
- Approximation of functions from Korobov spaces by shallow neural networks
- Deep spectral Q-learning with application to mobile health
- Compositional sparsity of learnable functions
- Characterizing 4-string contact interaction using machine learning
This page was built for publication: Nonparametric regression using deep neural networks with ReLU activation function
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2215715)
