Estimating prediction error for complex samples
From MaRDI portal
(Redirected from Publication:5094329)
Abstract: With a growing interest in using non-representative samples to train prediction models for numerous outcomes it is necessary to account for the sampling design that gives rise to the data in order to assess the generalized predictive utility of a proposed prediction rule. After learning a prediction rule based on a non-uniform sample, it is of interest to estimate the rule's error rate when applied to unobserved members of the population. Efron (1986) proposed a general class of covariance penalty inflated prediction error estimators that assume the available training data are representative of the target population for which the prediction rule is to be applied. We extend Efron's estimator to the complex sample context by incorporating Horvitz-Thompson sampling weights and show that it is consistent for the true generalization error rate when applied to the underlying superpopulation. The resulting Horvitz-Thompson-Efron (HTE) estimator is equivalent to dAIC, a recent extension of AIC to survey sampling data, but is more widely applicable. The proposed methodology is assessed with simulations and is applied to models predicting renal function obtained from the large-scale NHANES survey.
Recommendations
- Prediction error estimation of the survey-weighted least squares model under complex sampling
- Model-assisted survey estimation with modern prediction techniques
- Conditionally Consistent Estimators Using Only Probabilities of Selection in Complex Sample Surveys
- Model evaluation based on the sampling distribution of estimated absolute prediction error
- Nonparametric regression estimators in complex surveys
Cited in
(7)- Fast robust estimation of prediction error based on resampling
- Computer Experiments: Prediction Accuracy, Sample Size and Model Complexity Revisited
- Prediction error estimation of the survey-weighted least squares model under complex sampling
- Criterion constrained Bayesian hierarchical models
- K-fold cross-validation for complex sample surveys
- A covariance-based penalty estimator for model assessment with censored data
- Evaluating machine learning models in nonstandard settings: an overview and new findings
This page was built for publication: Estimating prediction error for complex samples
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5094329)
