On the minimax optimality and superiority of deep neural network learning over sparse parameter spaces

DOI10.1016/J.NEUNET.2019.12.014zbMATH Open1434.68508DBLPjournals/nn/HayakawaS20arXiv1905.09195OpenAlexW2996782006WikidataQ92414883 ScholiaQ92414883MaRDI QIDQ2185697FDOQ2185697

Authors: Satoshi Hayakawa, Taiji Suzuki 

Publication date: 5 June 2020

Published in: Neural Networks (Search for Journal in Brave)

Abstract: Deep learning has been applied to various tasks in the field of machine learning and has shown superiority to other common procedures such as kernel methods. To provide a better theoretical understanding of the reasons for its success, we discuss the performance of deep learning and other methods on a nonparametric regression problem with a Gaussian noise. Whereas existing theoretical studies of deep learning have been based mainly on mathematical theories of well-known function classes such as H"{o}lder and Besov classes, we focus on function classes with discontinuity and sparsity, which are those naturally assumed in practice. To highlight the effectiveness of deep learning, we compare deep learning with a class of linear estimators representative of a class of shallow estimators. It is shown that the minimax risk of a linear estimator on the convex hull of a target function class does not differ from that of the original target function class. This results in the suboptimality of linear methods over a simple but non-convex function class, on which deep learning can attain nearly the minimax-optimal rate. In addition to this extreme case, we consider function classes with sparse wavelet coefficients. On these function classes, deep learning also attains the minimax rate up to log factors of the sample size, and linear methods are still suboptimal if the assumed sparsity is strong. We also point out that the parameter sharing of deep neural networks can remarkably reduce the complexity of the model in our setting.

Full work available at URL: https://arxiv.org/abs/1905.09195

Recommendations

• Consistent Sparse Deep Learning: Theory and Computation
• Learning sparse deep neural networks with a spike-and-slab prior
• Nonparametric regression using deep neural networks with ReLU activation function
• Optimal nonparametric inference via deep neural network
• Deep learning: a statistical viewpoint

zbMATH Keywords

deep learningnonparametric regressionminimax optimalitylinear estimatorneural network

Mathematics Subject Classification ID

Nonparametric regression and quantile regression (62G08) Artificial neural networks and deep learning (68T07)

Cites Work

• The elements of statistical learning. Data mining, inference, and prediction
• Weak convergence and empirical processes. With applications to statistics
• Ideal spatial adaptation by wavelet shrinkage
• Pattern recognition and machine learning.
• Ten Lectures on Wavelets
• Title not available (Why is that?)
• Minimax Rates of Estimation for High-Dimensional Linear Regression Over $\ell_q$-Balls
• Local Rademacher complexities and oracle inequalities in risk minimization. (2004 IMS Medallion Lecture). (With discussions and rejoinder)
• Title not available (Why is that?)
• Wavelet threshold estimation of a regression function with random design
• Minimax estimation via wavelet shrinkage
• Concentration inequalities and asymptotic results for ratio type empirical processes
• Title not available (Why is that?)
• Approximation by superpositions of a sigmoidal function
• Minimax theory of image reconstruction
• Information-theoretic determination of minimax rates of convergence
• Minimax estimation of linear functionals over nonconvex parameter spaces.
• Minimax risk over hyperrectangles, and implications
• Unconditional bases and bit-level compression
• Unconditional bases are optimal bases for data compression and for statistical estimation
• Adaptive Minimax Estimation over Sparse $\ell_q$-Hulls
• Optimal approximation of piecewise smooth functions using deep ReLU neural networks
• Error bounds for approximations with deep ReLU networks
• Neural network with unbounded activation functions is universal approximator
• Nonparametric regression using deep neural networks with ReLU activation function

Cited In (9)

• Deep learning theory of distribution regression with CNNs
• Rejoinder: On nearly assumption-free tests of nominal confidence interval coverage for causal parameters estimated by machine learning
• Optimal nonparametric inference via deep neural network
• Estimation error analysis of deep learning on the regression problem on the variable exponent Besov space
• Adaptive deep learning for nonlinear time series models
• Consistent Sparse Deep Learning: Theory and Computation
• Title not available (Why is that?)
• Drift estimation for a multi-dimensional diffusion process using deep neural networks
• Title not available (Why is that?)

Uses Software

• PRMLT

This page was built for publication: On the minimax optimality and superiority of deep neural network learning over sparse parameter spaces

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2185697)