Statistical guarantees for regularized neural networks

DOI10.1016/J.NEUNET.2021.04.034MaRDI QIDQ6079063zbMATH OpenOpenAlexFDO

Authors M. Taheri, Fang Xie, Johannes Lederer

Publication date 28 September 2023

Published in Neural Networks (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2006.00294

zbMATH Keywords

deep learningregularizationneural networksprediction guarantees

Mathematics Subject Classification ID

Artificial neural networks and deep learning (68T07)

Abstract: Neural networks have become standard tools in the analysis of data, but they lack comprehensive mathematical theories. For example, there are very few statistical guarantees for learning neural networks from data, especially for classes of estimators that are used in practice or at least similar to such. In this paper, we develop a general statistical guarantee for estimators that consist of a least-squares term and a regularizer. We then exemplify this guarantee with $el{l}_1$-regularization, showing that the corresponding prediction error increases at most sub-linearly in the number of layers and at most logarithmically in the total number of parameters. Our results establish a mathematical basis for regularized estimation of neural networks, and they deepen our mathematical understanding of neural networks and deep learning more generally.

Recommendations

• Mean field analysis of neural networks: a law of large numbers
• Deep learning: a statistical viewpoint
• Learning theory and approximation by neural networks
• The Modern Mathematics of Deep Learning
• Error bounds for approximations using multichannel deep convolutional neural networks with downsampling

Cites work

• scientific article; zbMATH DE number 5654889 (Why is no real title available?)
• scientific article; zbMATH DE number 49190 (Why is no real title available?)
• scientific article; zbMATH DE number 845714 (Why is no real title available?)
• 10.1162/153244303321897690
• Compressed sensing
• Concentration inequalities. A nonasymptotic theory of independence
• Error bounds for approximations with deep ReLU networks
• Estimation and testing under sparsity. École d'Été de Probabilités de Saint-Flour XLV -- 2015
• High-dimensional probability. An introduction with applications in data science
• How Correlations Influence Lasso Prediction
• New concentration inequalities for suprema of empirical processes
• Nonparametric regression using deep neural networks with ReLU activation function
• On the prediction performance of the Lasso
• Oracle inequalities for high-dimensional prediction
• Stable signal recovery from incomplete and inaccurate measurements
• The Bernstein-Orlicz norm and deviation inequalities
• The sample complexity of pattern classification with neural networks: the size of the weights is more important than the size of the network
• Weak convergence and empirical processes. With applications to statistics

Cited in

(10)

• Guaranteed approximation error estimation of neural networks and model modification
• Regularization theory in the study of generalization ability of a biological neural network model
• Statistical guarantees for sparse deep learning
• A statistical model of neural network learning via the Cramer-Rao lower bound
• Mean field analysis of neural networks: a law of large numbers
• Layer sparsity in neural networks
• Searching for minimal optimal neural networks
• Function approximation by deep neural networks with parameters \(\{0, \pm \frac{1}{2}, \pm 1,2\}\)
• From kernel methods to neural networks: a unifying variational formulation
• Nonparametric regression with modified ReLU networks

This page was built for publication: Statistical guarantees for regularized neural networks

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6079063)