From Fixed-X to Random-X Regression: Bias-Variance Decompositions, Covariance Penalties, and Prediction Error Estimation
From MaRDI portal
Publication:3304841
Abstract: In statistical prediction, classical approaches for model selection and model evaluation based on covariance penalties are still widely used. Most of the literature on this topic is based on what we call the "Fixed-X" assumption, where covariate values are assumed to be nonrandom. By contrast, it is often more reasonable to take a "Random-X" view, where the covariate values are independently drawn for both training and prediction. To study the applicability of covariance penalties in this setting, we propose a decomposition of Random-X prediction error in which the randomness in the covariates contributes to both the bias and variance components. This decomposition is general, but we concentrate on the fundamental case of least squares regression. We prove that in this setting the move from Fixed-X to Random-X prediction results in an increase in both bias and variance. When the covariates are normally distributed and the linear model is unbiased, all terms in this decomposition are explicitly computable, which yields an extension of Mallows' Cp that we call . also holds asymptotically for certain classes of nonnormal covariates. When the noise variance is unknown, plugging in the usual unbiased estimate leads to an approach that we call , which is closely related to Sp (Tukey 1967), and GCV (Craven and Wahba 1978). For excess bias, we propose an estimate based on the "shortcut-formula" for ordinary cross-validation (OCV), resulting in an approach we call . Theoretical arguments and numerical simulations suggest that is typically superior to OCV, though the difference is small. We further examine the Random-X error of other popular estimators. The surprising result we get for ridge regression is that, in the heavily-regularized regime, Random-X variance is smaller than Fixed-X variance, which can lead to smaller overall Random-X error.
Recommendations
Cites work
- scientific article; zbMATH DE number 3154237 (Why is no real title available?)
- scientific article; zbMATH DE number 45848 (Why is no real title available?)
- scientific article; zbMATH DE number 1332320 (Why is no real title available?)
- scientific article; zbMATH DE number 3444596 (Why is no real title available?)
- scientific article; zbMATH DE number 3291685 (Why is no real title available?)
- A Biometrics Invited Paper. The Analysis and Selection of Variables in Linear Regression
- A comparative study of ordinary cross-validation, v-fold cross-validation and the repeated learning-testing methods
- A distribution-free theory of nonparametric regression
- A survey of cross-validation procedures for model selection
- Concentration of measure and spectra of random matrices: applications to correlation matrices, elliptical distributions and beyond
- Estimation of generalization error: random and fixed inputs
- Estimation of the mean of a multivariate normal distribution
- Evaluation and selection of models for out-of-sample prediction when the sample size is small relative to the complexity of the data-generating process
- Generalized Cross-Validation as a Method for Choosing a Good Ridge Parameter
- High-dimensional asymptotics of prediction: ridge regression and classification
- How Biased is the Apparent Error Rate of a Prediction Rule?
- How Many Variables Should be Entered in a Regression Equation?
- On the ``degrees of freedom of the lasso
- Optimal equivariant prediction for high-dimensional linear models with arbitrary predictor covariance
- Persistene in high-dimensional linear predictor-selection and the virtue of overparametrization
- Random design analysis of ridge regression
- Selection of Variables in Multiple Regression: Part I. A Review and Evaluation
- Selection of Variables in Multiple Regression: Part II. Chosen Procedures, Computations and Examples
- Smoothing noisy data with spline functions: Estimating the correct degree of smoothing by the method of generalized cross-validation
- Some Comments on C P
- The Estimation of Prediction Error
Cited in
(16)- Cross-Validation: What Does It Estimate and How Well Does It Do It?
- Empirical Bayes Mean Estimation With Nonparametric Errors Via Order Statistic Regression on Replicated Data
- Inference for a two-stage enrichment design
- Inadmissibility of the corrected Akaike information criterion
- Correcting the corrected AIC
- Prediction of the Nash through penalized mixture of logistic regression models
- Comment on: ``Models as approximations
- Semi-supervised empirical risk minimization: using unlabeled data to improve prediction
- From Fixed-X to Random-X Regression: Bias-Variance Decompositions, Covariance Penalties, and Prediction Error Estimation: Correction
- Active Operator Inference for Learning Low-Dimensional Dynamical-System Models from Noisy Data
- A mutual information criterion with applications to canonical correlation analysis and graphical models
- Optimal selection of sample-size dependent common subsets of covariates for multi-task regression prediction
- A Relabeling Approach to Handling the Class Imbalance Problem for Logistic Regression
- Homeostasis phenomenon in conformal prediction and predictive distribution functions
- Average effects based on regressions with a logarithmic link function: a new approach with stochastic covariates
- Fast and Exact Leave-One-Out Analysis of Large-Margin Classifiers
This page was built for publication: From Fixed-X to Random-X Regression: Bias-Variance Decompositions, Covariance Penalties, and Prediction Error Estimation
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q3304841)
