Two models of double descent for weak features
From MaRDI portal
Abstract: The "double descent" risk curve was proposed to qualitatively describe the out-of-sample prediction accuracy of variably-parameterized machine learning models. This article provides a precise mathematical analysis for the shape of this curve in two simple data models with the least squares/least norm predictor. Specifically, it is shown that the risk peaks when the number of features is close to the sample size , but also that the risk decreases towards its minimum as increases beyond . This behavior is contrasted with that of "prescient" models that select features in an a priori optimal order.
Recommendations
- Surprises in high-dimensional ridgeless least squares interpolation
- Reconciling modern machine-learning practice and the classical bias-variance trade-off
- A comparative analysis of optimization and generalization properties of two-layer neural network and random feature models under gradient descent dynamics
- Deep learning: a statistical viewpoint
- Linearized two-layers neural networks in high dimension
Cites work
- A jamming transition from under- to over-parametrization affects generalization in deep learning
- An elementary proof of a theorem of Johnson and Lindenstrauss
- Benign overfitting in linear regression
- Concentration inequalities for sampling without replacement
- Dynamics of batch training in a perceptron
- Generalization in a linear perceptron in the presence of noise
- High-dimensional probability. An introduction with applications in data science
- How Many Variables Should be Entered in a Regression Equation?
- Limiting empirical singular value distribution of restrictions of discrete Fourier transform matrices
- Reconciling modern machine-learning practice and the classical bias-variance trade-off
- Smallest singular value of a random rectangular matrix
Cited in
(55)- On the Inconsistency of Kernel Ridgeless Regression in Fixed Dimensions
- Regularized zero-variance control variates
- The interpolation phase transition in neural networks: memorization and generalization under lazy training
- Overparameterized maximum likelihood tests for detection of sparse vectors
- The generalization error of max-margin linear classifiers: benign overfitting and high dimensional asymptotics in the overparametrized regime
- Minimum discrepancy principle strategy for choosing k in k-NN regression
- On the robustness of the minimim _2 interpolator
- A Unifying Tutorial on Approximate Message Passing
- Learning curves of generic features maps for realistic datasets with a teacher-student model*
- Canonical thresholding for nonsparse high-dimensional linear regression
- Empirical risk minimization in the interpolating regime with application to neural network learning
- High-Dimensional Analysis of Double Descent for Linear Regression with Random Projections
- A geometrical viewpoint on the benign overfitting property of the minimum _2-norm interpolant estimator and its universality
- The common intuition to transfer learning can win or lose: case studies for linear regression
- Binary classification of Gaussian mixtures: abundance of support vectors, benign overfitting, and regularization
- Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation
- Theoretical issues in deep networks
- Magnitude and angle dynamics in training single ReLU neurons
- Reconciling modern machine-learning practice and the classical bias-variance trade-off
- Mehler’s Formula, Branching Process, and Compositional Kernels of Deep Neural Networks
- Overparameterization and generalization error: weighted trigonometric interpolation
- A Generalization Gap Estimation for Overparameterized Models via the Langevin Functional Variance
- Benign Overfitting and Noisy Features
- Large-dimensional random matrix theory and its applications in deep learning and wireless communications
- Differentially private learning beyond the classical dimensionality regime
- New equivalences between interpolation and SVMs: kernels and structured features
- Triple descent and the two kinds of overfitting: where and why do they appear?*
- High dimensional binary classification under label shift: phase transition and regularization
- scientific article; zbMATH DE number 7370646 (Why is no real title available?)
- scientific article; zbMATH DE number 7626737 (Why is no real title available?)
- High-dimensional regression with missing data: an asymptotic study
- Benign overfitting in linear regression
- Benefit of Interpolation in Nearest Neighbor Algorithms
- A model of double descent for high-dimensional binary linear classification
- Double Double Descent: On Generalization Errors in Transfer Learning between Linear Regression Tasks
- Conditioning of random Fourier feature matrices: double descent and generalization error
- Double data piling: a high-dimensional solution for asymptotically perfect multi-category classification
- A Universal Trade-off Between the Model Size, Test Loss, and Training Loss of Linear Predictors
- A dynamic view of some anomalous phenomena in SGD for over-parametrized neural networks
- Conservative SPDEs as fluctuating mean field limits of stochastic gradient descent
- Deep networks for system identification: a survey
- Generalization error rates in kernel regression: the crossover from the noiseless to noisy regime*
- MUSO: achieving exact machine unlearning in over-parameterized regimes
- scientific article; zbMATH DE number 7625163 (Why is no real title available?)
- Dimensionality Reduction, Regularization, and Generalization in Overparameterized Regressions
- Kurdyka-Łojasiewicz exponent via Hadamard parametrization
- Dropout drops double descent
- scientific article; zbMATH DE number 7415098 (Why is no real title available?)
- Approximate spectral decomposition of Fisher information matrix for simple ReLU networks
- How more data can hurt: instability and regularization in next-generation reservoir computing
- The high-dimensional asymptotics of principal component regression
- Dimension independent excess risk by stochastic gradient descent
- Deep double descent: where bigger models and more data hurt*
- Surprises in high-dimensional ridgeless least squares interpolation
- A precise high-dimensional asymptotic theory for boosting and minimum-\(\ell_1\)-norm interpolated classifiers
This page was built for publication: Two models of double descent for weak features
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5027013)
