Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC
From MaRDI portal
Abstract: Leave-one-out cross-validation (LOO) and the widely applicable information criterion (WAIC) are methods for estimating pointwise out-of-sample prediction accuracy from a fitted Bayesian model using the log-likelihood evaluated at the posterior simulations of the parameter values. LOO and WAIC have various advantages over simpler estimates of predictive error such as AIC and DIC but are less used in practice because they involve additional computational steps. Here we lay out fast and stable computations for LOO and WAIC that can be performed using existing simulation draws. We introduce an efficient computation of LOO using Pareto-smoothed importance sampling (PSIS), a new procedure for regularizing importance weights. Although WAIC is asymptotically equal to LOO, we demonstrate that PSIS-LOO is more robust in the finite case with weak priors or influential observations. As a byproduct of our calculations, we also obtain approximate standard errors for estimated predictive errors and for comparing of predictive errors between two models. We implement the computations in an R package called 'loo' and demonstrate using models fit with the Bayesian inference package Stan.
Recommendations
- Efficient leave-one-out cross-validation for Bayesian non-factorized normal and Student-t models
- scientific article; zbMATH DE number 6617268
- Approximating cross-validatory predictive evaluation in Bayesian latent variable models with integrated IS and WAIC
- Bayesian Model Assessment and Comparison Using Cross-Validation Predictive Densities
- Bayesian Measures of Model Complexity and Fit
Cites work
- scientific article; zbMATH DE number 6617268 (Why is no real title available?)
- scientific article; zbMATH DE number 3553528 (Why is no real title available?)
- scientific article; zbMATH DE number 578421 (Why is no real title available?)
- scientific article; zbMATH DE number 3444596 (Why is no real title available?)
- scientific article; zbMATH DE number 849928 (Why is no real title available?)
- A Predictive Approach to Model Selection
- A comparative study of ordinary cross-validation, v-fold cross-validation and the repeated learning-testing methods
- A survey of Bayesian predictive methods for model assessment, selection and comparison
- A survey of cross-validation procedures for model selection
- Asymptotic equivalence of Bayes cross validation and widely applicable information criterion in singular learning theory
- Bayesian Measures of Model Complexity and Fit
- Bayesian Model Assessment and Comparison Using Cross-Validation Predictive Densities
- Bayesian data analysis.
- Bayesian model averaging: A tutorial. (with comments and a rejoinder).
- Case-deletion importance sampling estimators: central limit theorems and related results
- Comparison of Bayesian predictive methods for model selection
- DIC in variable selection
- Erratum to: ``Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC
- Laplace approximation for logistic Gaussian process density estimation and regression
- On the Variability of Case-Deletion Importance Sampling Weights in the Bayesian Linear Model
- Penalized loss functions for Bayesian model comparison
- Strictly Proper Scoring Rules, Prediction, and Estimation
- Testing the assumptions behind importance sampling
- The no-U-turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo
- Understanding predictive information criteria for Bayesian models
Cited in
(only showing first 100 items - show all)- A new regression model for overdispersed binomial data accounting for outliers and an excess of zeros
- Monotonicity of rank order probabilities in signal detection models of simultaneous detection and identification
- Information criteria and cross validation for Bayesian inference in regular and singular cases
- Bayesian inference for partial orders from random linear extensions: power relations from 12th century royal acta
- An explanatory mixture <scp>IRT</scp> model for careless and insufficient effort responding in self‐report measures
- Exploring examinees' responses to constructed response items with a supervised topic model
- Extending exploratory diagnostic classification models: Inferring the effect of covariates
- Bayesian residual analysis for spatially correlated data
- The analysis of serve decisions in tennis using Bayesian hierarchical models
- Structured Shrinkage Priors
- Cross-Validatory Z-Residual for Diagnosing Shared Frailty Models
- Bayesian quantile regression models for heavy tailed bounded variables using the No-U-Turn sampler
- The quantile probability model
- Bayesian shared parameter joint models for heterogeneous populations
- A flexible procedure for formulating probability distributions on the unit interval with applications
- Generalized Bayes approach to inverse problems with model misspecification
- Prediction scoring of data-driven discoveries for reproducible research
- Contact data and SARS-CoV-2: retrospective analysis of the estimated impact of the first UK lockdown
- Item response and response time model for personality assessment via linear ballistic accumulation
- Detecting multiple random changepoints in Bayesian piecewise growth mixture models
- Bayesian inference and dynamic prediction for multivariate longitudinal and survival data
- Recent advances in algebraic geometry and Bayesian statistics
- Bayesian compositional generalized linear models for analyzing microbiome data
- Bayesian predictive model averaging approach to joint longitudinal-survival modeling: application to an immuno-oncology clinical trial
- Adaptive use of co-data through empirical Bayes for Bayesian additive regression trees
- Bayesian hierarchical penalized spline models for immediate and time-varying intervention effects in stepped wedge cluster randomized trials
- Prediction can be safely used as a proxy for explanation in causally consistent Bayesian generalized linear models
- Asymmetric exponential power Bayesian median autoregression with applications
- Computational Methods for Fast Bayesian Model Assessment via Calibrated Posterior p -values
- Approximating cross-validatory predictive evaluation in Bayesian latent variable models with integrated IS and WAIC
- Approximate leave-future-out cross-validation for Bayesian time series models
- Multidimensional graded response models with hierarchical structure and Q-matrix
- A Bayesian approach to model individual differences and to partition individuals: case studies in growth and learning curves
- Finite-dimensional Discrete Random Structures and Bayesian Clustering
- A multidimensional IRT model for ability-item-based guessing: the development of a two-parameter logistic extension model
- Longitudinal quantile-based regression models using multivariate asymmetric heavy-tailed distributions and leapfrog HMC algorithm
- Bayesian Multivariate Distributional Regression With Skewed Responses and Skewed Random Effects
- Bayesian model-averaged meta-analysis in medicine
- A hierarchical Bayesian state trace analysis for assessing monotonicity while factoring out subject, item, and trial level dependencies
- A modeler's guide to extreme value software
- High-dimensional modeling of spatial and spatio-temporal conditional extremes using INLA and Gaussian Markov random fields
- On outliers detection and prior distribution sensitivity in standard skew-probit regression models
- Projective inference in high-dimensional problems: prediction and feature selection
- Storvik, Palomares, Engebretsen, Rø, Engø-Monsen, Kristoffersen, De Blasio and Frigessi's reply to the discussion of `The second discussion meeting on statistical aspects of the COVID-19 pandemic'
- Bhatt, Ferguson, Flaxman, Gandy, Mishra, and Scott's reply to the discussion of `The second discussion meeting on statistical aspects of the COVID-19 pandemic'
- Jorge Mateu and Álvaro Briz-Redón's contribution to the discussion of `The second discussion meeting on statistical aspects of the COVID-19 pandemic'
- Andrew B. Lawson's contribution to the discussion of `The second discussion meeting on statistical aspects of the COVID-19 pandemic'
- Hans R. Künsch and Fabio Sigrist's contribution to the discussion of `The second discussion meeting on statistical aspects of the COVID-19 pandemic'
- Sawitree Boonpatcharanon, Jane Heffernan and Hanna Jankowski's contribution to the discussion of `The second discussion meeting on statistical aspects of the COVID-19 pandemic'
- Heejong Bong, Valerie Ventura and Larry Wasserman's contribution to the discussion of `The second discussion meeting on statistical aspects of the COVID-19 pandemic'
- Alice Corbella, Anne M. Presanis, Paul J. Birrell and Daniela de Angelis's contribution to the discussion of `The second discussion meeting on statistical aspects of the COVID-19 pandemic'
- Sanmitra Ghosh's contribution to the discussion of `The second discussion meeting on statistical aspects of the COVID-19 pandemic'
- Paul J. Birrell, Angelos Alexopoulos and Daniela de Angelis's contribution to the discussion of `The second discussion meeting on statistical aspects of the COVID-19 pandemic'
- Arun Chind's contribution to the discussion of `The second discussion meeting on statistical aspects of the COVID-19 pandemic'
- Seconder of the vote of thanks and contribution to the discussion of `The second discussion meeting on statistical aspects of the COVID-19 pandemic'
- Proposer of the vote of thanks and contribution to the discussion of `The second discussion meeting on statistical aspects of the COVID-19 pandemic'
- Semi-mechanistic Bayesian modelling of COVID-19 with renewal processes
- Disentangling positive and negative partisanship in social media interactions using a coevolving latent space network with attractors model
- Bayesian structured antedependence model proposals for longitudinal data
- Semiparametric regression for dual population mortality
- Bayesian inference for conditional copulas using Gaussian process single index models
- Cross-validatory model selection for Bayesian autoregressions with exogenous regressors
- Using stacking to average Bayesian predictive distributions (with discussion)
- Fundamental tools for developing likelihood functions within ACT-R
- Bayesian cross-validation by parallel Markov chain Monte Carlo
- Using degradation models to assess pipeline life
- Posterior covariance information criterion for weighted inference
- Transmission of macroeconomic shocks to risk parameters: their uses in stress testing
- PARSIMONIOUS PARAMETERIZATION OF AGE-PERIOD-COHORT MODELS BY BAYESIAN SHRINKAGE
- Performance of asymmetric links and correction methods for imbalanced data in binary regression
- Probabilistic approach to limited-data computed tomography reconstruction
- Bayesian hierarchical models for the prediction of volleyball results
- A Bayesian mixture model accounting for individual heterogeneity in response to pathogenic infection
- Support provided by elderly in Italy: a hierarchical analysis of ego networks controlling for alter-overlapping
- Learning low-dimensional structure in house price indices
- Efficient estimation and correction of selection-induced bias with order statistics
- The leave-worst-k-out criterion for cross validation
- Regression modelling with the tilted beta distribution: A Bayesian approach
- A Hierarchical Model for Heterogenous Reliability Field Data
- Comparisons of zero-augmented continuous regression models from a Bayesian perspective
- A Bayesian model of microbiome data for simultaneous identification of covariate associations and prediction of phenotypic outcomes
- Mapping ex ante risks of COVID-19 in Indonesia using a Bayesian geostatistical model on airport network data
- Poverty and inequality mapping based on a unit-level log-normal mixture model
- A hidden Markov space-time model for mapping the dynamics of global access to food
- Projection predictive variable selection for discrete response families with finite support
- Cross-cohort mixture analysis: a data integration approach with applications on gestational age and DNA-methylation-derived gestational age acceleration metrics
- Bayesian dynamic modelling to assess differential treatment effects on panic attack frequencies
- Model-based clustering of trends and cycles of nitrate concentrations in rivers across France
- Models to support forest inventory and small area estimation using sparsely sampled LiDAR: a case study involving G-LiHT LiDAR in Tanana, Alaska
- A Bayesian approach for the G-DINA model
- Unbiased estimator for the variance of the leave-one-out cross-validation estimator for a Bayesian normal model with fixed variance
- Development of a novel computational model for the balloon analogue risk task: the exponential-weight mean-variance model
- Inflection points in community-level homeless rates
- Bayesian mitigation of spatial coarsening for a Hawkes model applied to gunfire, wildfire and viral contagion
- Detecting and modeling changes in a time series of proportions
- Stochastic optimization with adaptive restart: a framework for integrated local and global learning
- An extended two-stage sequential optimization approach: properties and performance
- Bayesian modeling framework for optimizing pre-hospital stroke triage decisions
- Bayesian survival analysis of Rayleigh-X family with time varying covariate
- Bayesian modelling of effective and functional brain connectivity using hierarchical vector autoregressions
This page was built for publication: Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q59366)
