Statistical guarantees for regularized neural networks

DOI10.1016/J.NEUNET.2021.04.034zbMATH Open1521.68202arXiv2006.00294OpenAlexW3159120966MaRDI QIDQ6079063FDOQ6079063

Authors: M. Taheri, Fang Xie, Johannes Lederer 

Publication date: 28 September 2023

Published in: Neural Networks (Search for Journal in Brave)

Abstract: Neural networks have become standard tools in the analysis of data, but they lack comprehensive mathematical theories. For example, there are very few statistical guarantees for learning neural networks from data, especially for classes of estimators that are used in practice or at least similar to such. In this paper, we develop a general statistical guarantee for estimators that consist of a least-squares term and a regularizer. We then exemplify this guarantee with $el{l}_1$-regularization, showing that the corresponding prediction error increases at most sub-linearly in the number of layers and at most logarithmically in the total number of parameters. Our results establish a mathematical basis for regularized estimation of neural networks, and they deepen our mathematical understanding of neural networks and deep learning more generally.

Full work available at URL: https://arxiv.org/abs/2006.00294

Recommendations

• Mean field analysis of neural networks: a law of large numbers
• Deep learning: a statistical viewpoint
• Learning theory and approximation by neural networks
• The Modern Mathematics of Deep Learning
• Error bounds for approximations using multichannel deep convolutional neural networks with downsampling

zbMATH Keywords

deep learningregularizationneural networksprediction guarantees

Mathematics Subject Classification ID

Artificial neural networks and deep learning (68T07)

Cites Work

• Weak convergence and empirical processes. With applications to statistics
• Title not available (Why is that?)
• Estimation and testing under sparsity. École d'Été de Probabilités de Saint-Flour XLV -- 2015
• Title not available (Why is that?)
• High-dimensional probability. An introduction with applications in data science
• New concentration inequalities for suprema of empirical processes
• Concentration inequalities. A nonasymptotic theory of independence
• Title not available (Why is that?)
• Stable signal recovery from incomplete and inaccurate measurements
• Compressed sensing
• On the prediction performance of the Lasso
• 10.1162/153244303321897690
• The sample complexity of pattern classification with neural networks: the size of the weights is more important than the size of the network
• The Bernstein-Orlicz norm and deviation inequalities
• How Correlations Influence Lasso Prediction
• Error bounds for approximations with deep ReLU networks
• Oracle inequalities for high-dimensional prediction
• Nonparametric regression using deep neural networks with ReLU activation function

Cited In (10)

• Regularization theory in the study of generalization ability of a biological neural network model
• Statistical guarantees for sparse deep learning
• A statistical model of neural network learning via the Cramer-Rao lower bound
• Mean field analysis of neural networks: a law of large numbers
• Layer sparsity in neural networks
• Searching for minimal optimal neural networks
• Function approximation by deep neural networks with parameters \(\{0, \pm \frac{1}{2}, \pm 1,2\}\)
• From kernel methods to neural networks: a unifying variational formulation
• Nonparametric regression with modified ReLU networks
• Guaranteed approximation error estimation of neural networks and model modification

This page was built for publication: Statistical guarantees for regularized neural networks

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6079063)