From Fixed-X to Random-X Regression: Bias-Variance Decompositions, Covariance Penalties, and Prediction Error Estimation

DOI10.1080/01621459.2018.1424632zbMATH Open1437.62351arXiv1704.08160OpenAlexW2625919377MaRDI QIDQ3304841FDOQ3304841

Saharon Rosset, Ryan J. Tibshirani

Publication date: 3 August 2020

Published in: Journal of the American Statistical Association (Search for Journal in Brave)

Abstract: In statistical prediction, classical approaches for model selection and model evaluation based on covariance penalties are still widely used. Most of the literature on this topic is based on what we call the "Fixed-X" assumption, where covariate values are assumed to be nonrandom. By contrast, it is often more reasonable to take a "Random-X" view, where the covariate values are independently drawn for both training and prediction. To study the applicability of covariance penalties in this setting, we propose a decomposition of Random-X prediction error in which the randomness in the covariates contributes to both the bias and variance components. This decomposition is general, but we concentrate on the fundamental case of least squares regression. We prove that in this setting the move from Fixed-X to Random-X prediction results in an increase in both bias and variance. When the covariates are normally distributed and the linear model is unbiased, all terms in this decomposition are explicitly computable, which yields an extension of Mallows' Cp that we call $RCp$. $RCp$ also holds asymptotically for certain classes of nonnormal covariates. When the noise variance is unknown, plugging in the usual unbiased estimate leads to an approach that we call $hatRCp$, which is closely related to Sp (Tukey 1967), and GCV (Craven and Wahba 1978). For excess bias, we propose an estimate based on the "shortcut-formula" for ordinary cross-validation (OCV), resulting in an approach we call $RC{p}^{+}$. Theoretical arguments and numerical simulations suggest that $RC{P}^{+}$ is typically superior to OCV, though the difference is small. We further examine the Random-X error of other popular estimators. The surprising result we get for ridge regression is that, in the heavily-regularized regime, Random-X variance is smaller than Fixed-X variance, which can lead to smaller overall Random-X error.

Full work available at URL: https://arxiv.org/abs/1704.08160

Recommendations

• The Estimation of Prediction Error
• Random design analysis of ridge regression
• Variable selection for linear regression models with random covariates
• scientific article

zbMATH Keywords

bias-variance decompositioncovariance penaltiesrandom-X regression

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05) Ridge regression; shrinkage estimators (Lasso) (62J07) Analysis of variance and covariance (ANOVA) (62J10) Inference from stochastic processes and prediction (62M20)

Cites Work

• A comparative study of ordinary cross-validation, v-fold cross-validation and the repeated learning-testing methods
• Title not available (Why is that?)
• Some Comments on C P
• A survey of cross-validation procedures for model selection
• Estimation of the mean of a multivariate normal distribution
• Smoothing noisy data with spline functions: Estimating the correct degree of smoothing by the method of generalized cross-validation
• Generalized Cross-Validation as a Method for Choosing a Good Ridge Parameter
• Persistene in high-dimensional linear predictor-selection and the virtue of overparametrization
• Title not available (Why is that?)
• Title not available (Why is that?)
• How Biased is the Apparent Error Rate of a Prediction Rule?
• A Biometrics Invited Paper. The Analysis and Selection of Variables in Linear Regression
• Title not available (Why is that?)
• A distribution-free theory of nonparametric regression
• On the ``degrees of freedom of the lasso
• Evaluation and selection of models for out-of-sample prediction when the sample size is small relative to the complexity of the data-generating process
• The Estimation of Prediction Error
• Concentration of measure and spectra of random matrices: applications to correlation matrices, elliptical distributions and beyond
• Random design analysis of ridge regression
• Optimal equivariant prediction for high-dimensional linear models with arbitrary predictor covariance
• How Many Variables Should be Entered in a Regression Equation?
• Estimation of generalization error: random and fixed inputs
• Selection of Variables in Multiple Regression: Part II. Chosen Procedures, Computations and Examples
• Selection of Variables in Multiple Regression: Part I. A Review and Evaluation
• Title not available (Why is that?)
• High-dimensional asymptotics of prediction: ridge regression and classification

Cited In (16)

• Cross-Validation: What Does It Estimate and How Well Does It Do It?
• Average effects based on regressions with a logarithmic link function: a new approach with stochastic covariates
• Empirical Bayes Mean Estimation With Nonparametric Errors Via Order Statistic Regression on Replicated Data
• Correcting the corrected AIC
• Prediction of the Nash through penalized mixture of logistic regression models
• Semi-supervised empirical risk minimization: using unlabeled data to improve prediction
• Active Operator Inference for Learning Low-Dimensional Dynamical-System Models from Noisy Data
• A Relabeling Approach to Handling the Class Imbalance Problem for Logistic Regression
• Inadmissibility of the corrected Akaike information criterion
• Inference for a two-stage enrichment design
• From Fixed-X to Random-X Regression: Bias-Variance Decompositions, Covariance Penalties, and Prediction Error Estimation: Correction
• Homeostasis phenomenon in conformal prediction and predictive distribution functions
• Optimal selection of sample-size dependent common subsets of covariates for multi-task regression prediction
• Comment on: ``Models as approximations
• A mutual information criterion with applications to canonical correlation analysis and graphical models
• Fast and Exact Leave-One-Out Analysis of Large-Margin Classifiers

Uses Software

• ElemStatLearn

This page was built for publication: From Fixed-X to Random-X Regression: Bias-Variance Decompositions, Covariance Penalties, and Prediction Error Estimation

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q3304841)