Prediction error after model search

DOI10.1214/19-AOS1818MaRDI QIDQ2196193zbMATH OpenOpenAlexFDO

Authors Xiaoying Tian

Publication date 28 August 2020

Published in The Annals of Statistics (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1610.06107, https://projecteuclid.org/euclid.aos/1590480033

zbMATH Keywords

prediction errordegrees of freedommodel searchStein's unbiased risk estimator (SURE)

Mathematics Subject Classification ID

Asymptotic properties of parametric estimators (62F12) Statistical ranking and selection procedures (62F07) Estimation in multivariate analysis (62H12) Ridge regression; shrinkage estimators (Lasso) (62J07) Inference from stochastic processes and prediction (62M20)

Abstract: Estimation of the prediction error of a linear estimation rule is difficult if the data analyst also use data to select a set of variables and construct the estimation rule using only the selected variables. In this work, we propose an asymptotically unbiased estimator for the prediction error after model search. Under some additional mild assumptions, we show that our estimator converges to the true prediction error in $L^2$ at the rate of $O(n^{-1/2})$, with $n$ being the number of data points. Our estimator applies to general selection procedures, not requiring analytical forms for the selection. The number of variables to select from can grow as an exponential factor of $n$, allowing applications in high-dimensional data. It also allows model misspecifications, not requiring linear underlying models. One application of our method is that it provides an estimator for the degrees of freedom for many discontinuous estimation rules like best subset selection or relaxed Lasso. Connection to Stein's Unbiased Risk Estimator is discussed. We consider in-sample prediction errors in this work, with some extension to out-of-sample errors in low dimensional, linear models. Examples such as best subset selection and relaxed Lasso are considered in simulations, where our estimator outperforms both $C_p$ and cross validation in various settings.

Recommendations

• Model imperfection and predicting predictability
• Estimation and accuracy after model selection
• Model selection and error estimation
• Publication:5750147
• Model error propagation from experimental to prediction configuration
• Decomposition of Prediction Error
• scientific article; zbMATH DE number 522884
• Relative-error prediction

Cites work

• scientific article; zbMATH DE number 845714 (Why is no real title available?)
• A note on an inequality involving the normal distribution
• A study of error variance estimation in Lasso regression
• A unified framework for high-dimensional analysis of \(M\)-estimators with decomposable regularizers
• Adaptive Model Selection
• Adaptive estimation of a quadratic functional by model selection.
• An inequality for the multivariate normal distribution
• Asymptotics of selective inference
• Can the strengths of AIC and BIC be shared? A conflict between model indentification and regression estimation
• Degrees of freedom and model search
• Degrees of freedom for piecewise Lipschitz estimators
• Degrees of freedom in lasso problems
• Differential Privacy: A Survey of Results
• Estimation of the mean of a multivariate normal distribution
• Honest confidence regions for nonparametric regression
• Least angle regression. (With discussion)
• On Measuring and Correcting the Effects of Data Mining and Model Selection
• On the ``degrees of freedom of the lasso
• Relaxed Lasso
• Scaled sparse linear regression
• Selective inference with a randomized response
• Selective inference with unknown variance via the square-root Lasso
• Simultaneous analysis of Lasso and Dantzig selector
• Some Comments on C P
• The Estimation of Prediction Error

Cited in

(4)

• Excess optimism: how biased is the apparent error of an estimator tuned by SURE?
• On the estimation of prediction errors in linear regression models
• Prediction errors for penalized regressions based on generalized approximate message passing
• Confidently Comparing Estimates with the c-value

Describes a project that uses

Uses Software

• selectiveInference

This page was built for publication: Prediction error after model search

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2196193)