Consistent Risk Estimation in Moderately High-Dimensional Linear Regression

DOI10.1109/TIT.2021.3095375MaRDI QIDQ4958281zbMATH OpenOpenAlexFDO

Authors Ji Xu, Arian Maleki, Kamiar Rahnama Rad, Daniel Hsu

Publication date 7 September 2021

Published in IEEE Transactions on Information Theory (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1902.01753

zbMATH Keywords

consistency of risk estimators

Mathematics Subject Classification ID

Linear regression; mixed models (62J05)

Abstract: Risk estimation is at the core of many learning systems. The importance of this problem has motivated researchers to propose different schemes, such as cross validation, generalized cross validation, and Bootstrap. The theoretical properties of such estimates have been extensively studied in the low-dimensional settings, where the number of predictors $p$ is much smaller than the number of observations $n$. However, a unifying methodology accompanied with a rigorous theory is lacking in high-dimensional settings. This paper studies the problem of risk estimation under the moderately high-dimensional asymptotic setting $n,pightarrowinfty$ and $n/pightarrowdelta>1$ ($delta$ is a fixed number), and proves the consistency of three risk estimates that have been successful in numerical studies, i.e., leave-one-out cross validation (LOOCV), approximate leave-one-out (ALO), and approximate message passing (AMP)-based techniques. A corner stone of our analysis is a bound that we obtain on the discrepancy of the `residuals' obtained from AMP and LOOCV. This connection not only enables us to obtain a more refined information on the estimates of AMP, ALO, and LOOCV, but also offers an upper bound on the convergence rate of each estimate.

Cited in

(7)

• The risk inflation criterion for multiple regression
• Unbiased predictive risk estimation of the Tikhonov regularization parameter: convergence with increasing rank approximations of the singular value decomposition
• Surprises in high-dimensional ridgeless least squares interpolation
• Augmented-limited regression models with an application to the study of the risk perceived using continuous scales
• A tradeoff between false discovery and true positive proportions for sparse high-dimensional logistic regression
• Corrected generalized cross-validation for finite ensembles of penalized estimators
• A new central limit theorem for the augmented IPW estimator: variance inflation, cross-fit covariance and beyond

This page was built for publication: Consistent Risk Estimation in Moderately High-Dimensional Linear Regression

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q4958281)