Semi-Supervised Approaches to Efficient Evaluation of Model Prediction Performance
From MaRDI portal
Abstract: In many modern machine learning applications, the outcome is expensive or time-consuming to collect while the predictor information is easy to obtain. Semi-supervised learning (SSL) aims at utilizing large amounts of `unlabeled' data along with small amounts of `labeled' data to improve the efficiency of a classical supervised approach. Though numerous SSL classification and prediction procedures have been proposed in recent years, no methods currently exist to evaluate the prediction performance of a working regression model. In the context of developing phenotyping algorithms derived from electronic medical records (EMR), we present an efficient two-step estimation procedure for evaluating a binary classifier based on various prediction performance measures in the semi-supervised (SS) setting. In step I, the labeled data is used to obtain a non-parametrically calibrated estimate of the conditional risk function. In step II, SS estimates of the prediction accuracy parameters are constructed based on the estimated conditional risk function and the unlabeled data. We demonstrate that under mild regularity conditions the proposed estimators are consistent and asymptotically normal. Importantly, the asymptotic variance of the SS estimators is always smaller than that of the supervised counterparts under correct model specification. We also correct for potential overfitting bias in the SS estimators in finite sample with cross-validation and develop a perturbation resampling procedure to approximate their distributions. Our proposals are evaluated through extensive simulation studies and illustrated with two real EMR studies aiming to develop phenotyping algorithms for rheumatoid arthritis and multiple sclerosis.
Recommendations
Cited in
(17)- A Bayesian precision medicine framework for calibrating individualized therapeutic indices in cancer
- Semi-supervised validation of multiple surrogate outcomes with application to electronic medical records phenotyping
- Automated feature selection of predictors in electronic medical records data
- Comparaison de différentes approches de l'évaluation supervisée
- Robust and efficient semi‐supervised estimation of average treatment effects with application to electronic health records data
- Semi-supervised inference for case-control binary data under possibly mis-specified logistic models
- A General M-estimation Theory in Semi-Supervised Framework
- Semi-supervised inference for nonparametric logistic regression
- Design and analysis of nested case-control studies for recurrent events subject to a terminal event
- Semi-supervised inference for the high-dimensional quantile regression
- Semi-supervised U-statistics
- Group inference of high-dimensional single-index models
- Solving the missing at random problem in semi-supervised learning: an inverse probability weighting method
- Efficient semiparametric estimation in two-sample comparison via semisupervised learning
- Doubly Robust Augmented Model Accuracy Transfer Inference with High Dimensional Features
- Leveraging Unlabeled Data for Superior ROC Curve Estimation via a Semiparametric Approach
- Generative semi-supervised classification
This page was built for publication: Semi-Supervised Approaches to Efficient Evaluation of Model Prediction Performance
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q4962081)
