Prediction scoring of data-driven discoveries for reproducible research
From MaRDI portal
Publication:2104015
DOI10.1007/S11222-022-10154-7zbMATH Open1499.62035arXiv2211.10314OpenAlexW4310579044MaRDI QIDQ2104015FDOQ2104015
Authors: Anna L. Smith, Tian Zheng, Andrew Gelman 
Publication date: 9 December 2022
Published in: Statistics and Computing (Search for Journal in Brave)
Abstract: Predictive modeling uncovers knowledge and insights regarding a hypothesized data generating mechanism (DGM). Results from different studies on a complex DGM, derived from different data sets, and using complicated models and algorithms, are hard to quantitatively compare due to random noise and statistical uncertainty in model results. This has been one of the main contributors to the replication crisis in the behavioral sciences. The contribution of this paper is to apply prediction scoring to the problem of comparing two studies, such as can arise when evaluating replications or competing evidence. We examine the role of predictive models in quantitatively assessing agreement between two datasets that are assumed to come from two distinct DGMs. We formalize a distance between the DGMs that is estimated using cross validation. We argue that the resulting prediction scores depend on the predictive models created by cross validation. In this sense, the prediction scores measure the distance between DGMs, along the dimension of the particular predictive model. Using human behavior data from experimental economics, we demonstrate that prediction scores can be used to evaluate preregistered hypotheses and provide insights comparing data from different populations and settings. We examine the asymptotic behavior of the prediction scores using simulated experimental data and demonstrate that leveraging competing predictive models can reveal important differences between underlying DGMs. Our proposed cross-validated prediction scores are capable of quantifying differences between unobserved data generating mechanisms and allow for the validation and assessment of results from complex models.
Full work available at URL: https://arxiv.org/abs/2211.10314
Recommendations
- Information confidence scores for prediction models
- Bayesian nonparametric cross-study validation of prediction methods
- Training replicable predictors in multiple studies
- The lack of cross-validation can lead to inflated results and spurious conclusions: a re-analysis of the MacArthur violence risk assessment study
- Statistical inference for measures of predictive success
Cites Work
- The elements of statistical learning. Data mining, inference, and prediction
- Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC
- Measuring and testing dependence by correlation of distances
- A survey of Bayesian predictive methods for model assessment, selection and comparison
- Understanding predictive information criteria for Bayesian models
- Strictly Proper Scoring Rules, Prediction, and Estimation
- Probabilistic Forecasts, Calibration and Sharpness
- Remarks on a Multivariate Transformation
- The ASA Statement on p-Values: Context, Process, and Purpose
- Consistent cross-validatory model-selection for dependent data: hv-block cross-validation
- Title not available (Why is that?)
- Title not available (Why is that?)
- Difficulty of selecting among multilevel models using predictive accuracy
- ON A METHOD OF DETERMINING WHETHER A SAMPLE OF SIZE n SUPPOSED TO HAVE BEEN DRAWN FROM A PARENT POPULATION HAVING A KNOWN PROBABILITY INTEGRAL HAS PROBABLY BEEN DRAWN AT RANDOM
- Conditional vs marginal estimation of the predictive loss of hierarchical models using WAIC and cross-validation
- Approximating cross-validatory predictive evaluation in Bayesian latent variable models with integrated IS and WAIC
- Data analysis, computation and mathematics
- Predictive Inference and Scientific Reproducibility
Cited In (2)
This page was built for publication: Prediction scoring of data-driven discoveries for reproducible research
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2104015)
