Optimal learning

DOI10.1007/S10092-023-00564-YMaRDI QIDQ6151544zbMATH OpenFDO

Authors Peter Binev, Andrea Bonito, Ronald DeVore, Guergana Petrova

Publication date 11 March 2024

Published in Calcolo (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2203.15994

zbMATH Keywords

regularizationBanach spaceChebyshev radiusoptimal learningover-parametrization

Mathematics Subject Classification ID

Linear regression; mixed models (62J05) Learning and adaptive systems in artificial intelligence (68T05) Numerical solutions of ill-posed problems in abstract spaces; regularization (65J20)

Abstract: This paper studies the problem of learning an unknown function $f$ from given data about $f$. The learning problem is to give an approximation $hatf$ to $f$ that predicts the values of $f$ away from the data. There are numerous settings for this learning problem depending on (i) what additional information we have about $f$ (known as a model class assumption), (ii) how we measure the accuracy of how well $hatf$ predicts $f$, (iii) what is known about the data and data sites, (iv) whether the data observations are polluted by noise. A mathematical description of the optimal performance possible (the smallest possible error of recovery) is known in the presence of a model class assumption. Under standard model class assumptions, it is shown in this paper that a near optimal $hatf$ can be found by solving a certain discrete over-parameterized optimization problem with a penalty term. Here, near optimal means that the error is bounded by a fixed constant times the optimal error. This explains the advantage of over-parameterization which is commonly used in modern machine learning. The main results of this paper prove that over-parameterized learning with an appropriate loss function gives a near optimal approximation $hatf$ of the function $f$ from which the data is collected. Quantitative bounds are given for how much over-parameterization needs to be employed and how the penalization needs to be scaled in order to guarantee a near optimal recovery of $f$. An extension of these results to the case where the data is polluted by additive deterministic noise is also given.

Recommendations

• The Modern Mathematics of Deep Learning
• Learning the kernel function via regularization
• From inexact optimization to learning via gradient concentration
• Overparameterization and generalization error: weighted trigonometric interpolation
• Optimal estimators in learning theory

Cites work

• scientific article; zbMATH DE number 3688714 (Why is no real title available?)
• scientific article; zbMATH DE number 3601500 (Why is no real title available?)
• scientific article; zbMATH DE number 713342 (Why is no real title available?)
• scientific article; zbMATH DE number 845714 (Why is no real title available?)
• scientific article; zbMATH DE number 6438182 (Why is no real title available?)
• A mathematical introduction to compressive sensing
• A new upper bound for sampling numbers
• A unifying representer theorem for inverse problems and machine learning
• Banach space representer theorems for neural networks and ridge splines
• Best subset, forward stepwise or Lasso? Analysis and recommendations based on extensive comparisons
• Compressed sensing
• Compressed sensing and best \(k\)-term approximation
• Correcting for unknown errors in sparse high-dimensional function approximation
• Data assimilation and sampling in Banach spaces
• Estimation and testing under sparsity. École d'Été de Probabilités de Saint-Flour XLV -- 2015
• Function values are enough for \(L_2\)-approximation
• Lasso-type recovery of sparse representations for high-dimensional data
• Minimization of functions having Lipschitz continuous first partial derivatives
• Neural network approximation
• On the stability and accuracy of least squares approximations
• Robust instance-optimal recovery of sparse signals at unknown noise levels
• Simultaneous analysis of Lasso and Dantzig selector
• Sobolev bounds on functions with scattered zeros, with applications to radial basis function surface fitting
• Sparse Polynomial Approximation of High-Dimensional Functions
• Square-root lasso: pivotal recovery of sparse signals via conic programming
• The Group Square-Root Lasso: Theoretical Properties and Fast Algorithms
• The sparsity of LASSO-type minimizers
• Tractability of multivariate problems. Volume I: Linear information
• Universal approximation bounds for superpositions of a sigmoidal function

Cited in

(3)

• Weighted variation spaces and approximation by shallow ReLU networks
• Applied harmonic analysis and data science. Abstracts from the workshop held April 21--26, 2024
• Optimal recovery from inaccurate data in Hilbert spaces: regularize, but what of the parameter?

This page was built for publication: Optimal learning

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6151544)