Prediction error after model search

DOI10.1214/19-AOS1818zbMATH Open1448.62137arXiv1610.06107OpenAlexW3030749005MaRDI QIDQ2196193FDOQ2196193

Authors: Xiaoying Tian 

Publication date: 28 August 2020

Published in: The Annals of Statistics (Search for Journal in Brave)

Abstract: Estimation of the prediction error of a linear estimation rule is difficult if the data analyst also use data to select a set of variables and construct the estimation rule using only the selected variables. In this work, we propose an asymptotically unbiased estimator for the prediction error after model search. Under some additional mild assumptions, we show that our estimator converges to the true prediction error in $L^2$ at the rate of $O(n^{-1/2})$, with $n$ being the number of data points. Our estimator applies to general selection procedures, not requiring analytical forms for the selection. The number of variables to select from can grow as an exponential factor of $n$, allowing applications in high-dimensional data. It also allows model misspecifications, not requiring linear underlying models. One application of our method is that it provides an estimator for the degrees of freedom for many discontinuous estimation rules like best subset selection or relaxed Lasso. Connection to Stein's Unbiased Risk Estimator is discussed. We consider in-sample prediction errors in this work, with some extension to out-of-sample errors in low dimensional, linear models. Examples such as best subset selection and relaxed Lasso are considered in simulations, where our estimator outperforms both $C_p$ and cross validation in various settings.

Full work available at URL: https://arxiv.org/abs/1610.06107

Recommendations

• Model imperfection and predicting predictability
• Estimation and accuracy after model selection
• Model selection and error estimation
• Publication:5750147
• Model error propagation from experimental to prediction configuration
• Decomposition of Prediction Error
• scientific article; zbMATH DE number 522884
• Relative-error prediction

zbMATH Keywords

prediction errordegrees of freedommodel searchStein's unbiased risk estimator (SURE)

Mathematics Subject Classification ID

Asymptotic properties of parametric estimators (62F12) Statistical ranking and selection procedures (62F07) Estimation in multivariate analysis (62H12) Ridge regression; shrinkage estimators (Lasso) (62J07) Inference from stochastic processes and prediction (62M20)

Cites Work

• Least angle regression. (With discussion)
• Title not available (Why is that?)
• Some Comments on C P
• Estimation of the mean of a multivariate normal distribution
• Simultaneous analysis of Lasso and Dantzig selector
• A study of error variance estimation in Lasso regression
• A unified framework for high-dimensional analysis of \(M\)-estimators with decomposable regularizers
• Honest confidence regions for nonparametric regression
• Scaled sparse linear regression
• Can the strengths of AIC and BIC be shared? A conflict between model indentification and regression estimation
• Degrees of freedom in lasso problems
• Relaxed Lasso
• Selective inference with a randomized response
• On Measuring and Correcting the Effects of Data Mining and Model Selection
• Asymptotics of selective inference
• Adaptive estimation of a quadratic functional by model selection.
• On the ``degrees of freedom of the lasso
• Differential Privacy: A Survey of Results
• The Estimation of Prediction Error
• A note on an inequality involving the normal distribution
• An inequality for the multivariate normal distribution
• Adaptive Model Selection
• Degrees of freedom for piecewise Lipschitz estimators
• Degrees of freedom and model search
• Selective inference with unknown variance via the square-root Lasso

Cited In (4)

• Confidently Comparing Estimates with the c-value
• On the estimation of prediction errors in linear regression models
• Excess optimism: how biased is the apparent error of an estimator tuned by SURE?
• Prediction errors for penalized regressions based on generalized approximate message passing

Uses Software

• selectiveInference

This page was built for publication: Prediction error after model search

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2196193)