Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC
From MaRDI portal
Abstract: Leave-one-out cross-validation (LOO) and the widely applicable information criterion (WAIC) are methods for estimating pointwise out-of-sample prediction accuracy from a fitted Bayesian model using the log-likelihood evaluated at the posterior simulations of the parameter values. LOO and WAIC have various advantages over simpler estimates of predictive error such as AIC and DIC but are less used in practice because they involve additional computational steps. Here we lay out fast and stable computations for LOO and WAIC that can be performed using existing simulation draws. We introduce an efficient computation of LOO using Pareto-smoothed importance sampling (PSIS), a new procedure for regularizing importance weights. Although WAIC is asymptotically equal to LOO, we demonstrate that PSIS-LOO is more robust in the finite case with weak priors or influential observations. As a byproduct of our calculations, we also obtain approximate standard errors for estimated predictive errors and for comparing of predictive errors between two models. We implement the computations in an R package called 'loo' and demonstrate using models fit with the Bayesian inference package Stan.
Recommendations
- Efficient leave-one-out cross-validation for Bayesian non-factorized normal and Student-t models
- scientific article; zbMATH DE number 6617268
- Approximating cross-validatory predictive evaluation in Bayesian latent variable models with integrated IS and WAIC
- Bayesian Model Assessment and Comparison Using Cross-Validation Predictive Densities
- Bayesian Measures of Model Complexity and Fit
Cites work
- scientific article; zbMATH DE number 6617268 (Why is no real title available?)
- scientific article; zbMATH DE number 3553528 (Why is no real title available?)
- scientific article; zbMATH DE number 578421 (Why is no real title available?)
- scientific article; zbMATH DE number 3444596 (Why is no real title available?)
- scientific article; zbMATH DE number 849928 (Why is no real title available?)
- A Predictive Approach to Model Selection
- A comparative study of ordinary cross-validation, v-fold cross-validation and the repeated learning-testing methods
- A survey of Bayesian predictive methods for model assessment, selection and comparison
- A survey of cross-validation procedures for model selection
- Asymptotic equivalence of Bayes cross validation and widely applicable information criterion in singular learning theory
- Bayesian Measures of Model Complexity and Fit
- Bayesian Model Assessment and Comparison Using Cross-Validation Predictive Densities
- Bayesian data analysis.
- Bayesian model averaging: A tutorial. (with comments and a rejoinder).
- Case-deletion importance sampling estimators: central limit theorems and related results
- Comparison of Bayesian predictive methods for model selection
- DIC in variable selection
- Erratum to: ``Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC
- Laplace approximation for logistic Gaussian process density estimation and regression
- On the Variability of Case-Deletion Importance Sampling Weights in the Bayesian Linear Model
- Penalized loss functions for Bayesian model comparison
- Strictly Proper Scoring Rules, Prediction, and Estimation
- Testing the assumptions behind importance sampling
- The no-U-turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo
- Understanding predictive information criteria for Bayesian models
Cited in
(only showing first 100 items - show all)- Bayesian Model Assessment and Comparison Using Cross-Validation Predictive Densities
- Bayesian Non-Parametric Factor Analysis for Longitudinal Spatial Surfaces
- Identifying and interpreting subgroups in health care utilization data with count mixture regression models
- Embedded multilevel regression and poststratification: model-based inference with incomplete auxiliary information
- Using leave-one-out cross validation (LOO) in a multilevel regression and poststratification (MRP) workflow: a cautionary tale
- Robust Leave-One-Out Cross-Validation for High-Dimensional Bayesian Models
- Sensitivity to Unobserved Confounding in Studies with Factor-Structured Outcomes
- Bayesian estimation of a flexible bifactor generalized partial credit model to survey data
- Quantifying conditional probability tables in Bayesian networks: Bayesian regression for scenario-based encoding of elicited expert assessments on feral pig habitat
- Hierarchical Bayesian models of reinforcement learning: introduction and comparison to alternative methods
- Dynamic response strategies: accounting for response process heterogeneity in irtree decision nodes
- Analysis of single-index models with scale mixture of normals errors by using Bayesian P-splines
- Massive Parallelization Boosts Big Bayesian Multidimensional Scaling
- How close and how much? Linking health outcomes to built environment spatial distributions
- scientific article; zbMATH DE number 6617268 (Why is no real title available?)
- Bayesian hierarchical stacking: some models are (somewhere) useful
- Controlling the flexibility of non-Gaussian processes through shrinkage priors
- Bayesian functional registration of fMRI activation maps
- Modelling publication bias and p-hacking
- A Bayesian Approach to Envelope Quantile Regression
- Neuro-cognitive models of single-trial EEG measures describe latent effects of spatial attention during perceptual decision making
- A spatial mixed-effects regression model for electoral data
- Bayesian spatial modeling for housing data in South Africa
- The reliability factor: modeling individual reliability with multiple items from a single assessment
- A Bayesian generalized explanatory item response model to account for learning during the test
- Practical Hilbert space approximate Bayesian Gaussian processes for probabilistic programming
- Bayesian Variable Selection for Gaussian Copula Regression Models
- Robustness against outliers: A new variance inflated regression model for proportions
- Weighted leave-one-out cross validation
- Finite population survey sampling: an unapologetic Bayesian perspective
- Quantification of empirical determinacy: The impact of likelihood weighting on posterior location and spread in Bayesian meta-analysis estimated with JAGS and INLA
- Bayesian Approaches to Shrinkage and Sparse Estimation
- Optimal group decision: a matter of confidence calibration
- History and nature of the Jeffreys-Lindley paradox
- Transfer of macroeconomic shocks in stress tests modeling
- A Bayesian spatial model for imaging genetics
- Bayesian analysis of first-order Markov models for autocorrelated binary responses
- R-squared for Bayesian Regression Models
- Fitting Latent Non-Gaussian Models Using Variational Bayes and Laplace Approximations
- Bayesian hierarchical models for the combination of spatially misaligned data: a comparison of melding and downscaler approaches using INLA and SPDE
- BHAFT: Bayesian heredity-constrained accelerated failure time models for detecting gene-environment interactions in survival analysis
- An augmented illness-death model for semi-competing risks with clinically immediate terminal events
- Bayesian inference and model comparison for metallic fatigue data
- Bayesian clustering of spatial functional data with application to a human mobility study during COVID-19
- Estimating the stillbirth rate for 195 countries using a Bayesian sparse regression model with temporal smoothing
- Using reference models in variable selection
- Bayesian inference for an unknown number of attributes in restricted latent class models
- Bayesian extension of the Weibull AFT shared frailty model with generalized family of distributions for enhanced survival analysis using censored data
- Vector time series modelling of turbidity in Dublin Bay
- Marginal Likelihood Computation for Model Selection and Hypothesis Testing: An Extensive Review
- Multivariate Conway-Maxwell-Poisson Distribution: Sarmanov Method and Doubly Intractable Bayesian Inference
- Hierarchical Bayes modelling of penalty conversion rates of bundesliga players
- A Bayesian time-varying effect model for behavioral mHealth data
- Region-referenced spectral power dynamics of EEG signals: a hierarchical modeling approach
- Bayesian multivariate sparse functional principal components analysis with application to longitudinal microbiome multiomics data
- A two-stage Bayesian small area estimation approach for proportions
- A spatially varying hierarchical random effects model for longitudinal macular structural data in glaucoma patients
- Extended beta models for poverty mapping. An application integrating survey and remote sensing data in Bangladesh
- Bayesian parameter inference for epithelial mechanics
- Conditional vs marginal estimation of the predictive loss of hierarchical models using WAIC and cross-validation
- Fast and accurate estimation of non-nested binomial hierarchical models using variational inference
- Bayesian mixed effects models for zero-inflated compositions in microbiome data analysis
- Extending RT-MPTs to enable equal process times
- Modeling COVID-19 pandemic using Bayesian analysis with application to Slovene data
- Bayesian model-based clustering for longitudinal ordinal data
- Bayesian cylindrical data modeling using Abe-Ley mixtures
- Bayesian comparison of latent variable models: conditional versus marginal likelihoods
- Joint modeling of distances and times in point-count surveys
- Modeling obesity rate with spatial auto-correlation: a case study
- Bayesian mixture model of extended redundancy analysis
- A skew-normal Bayesian semi-parametric latent trait linear mixed effect model
- Past, present and future of software for Bayesian inference
- Bayesian model selection in the \(\mathcal{M}\)-open setting -- approximate posterior inference and subsampling for efficient large-scale leave-one-out cross-validation via the difference estimator
- Parameter continuity in time-varying Gauss-Markov models for learning from small training data sets
- A general approximation to nested Bayes factors with informed priors
- Bayesian modelling of exponentiated Weibull generated family for interval-censored data with rstan
- A new regression model for overdispersed binomial data accounting for outliers and an excess of zeros
- Monotonicity of rank order probabilities in signal detection models of simultaneous detection and identification
- Information criteria and cross validation for Bayesian inference in regular and singular cases
- An explanatory mixture <scp>IRT</scp> model for careless and insufficient effort responding in self‐report measures
- Exploring examinees' responses to constructed response items with a supervised topic model
- Extending exploratory diagnostic classification models: Inferring the effect of covariates
- Bayesian residual analysis for spatially correlated data
- The analysis of serve decisions in tennis using Bayesian hierarchical models
- Structured Shrinkage Priors
- The quantile probability model
- A flexible procedure for formulating probability distributions on the unit interval with applications
- Generalized Bayes approach to inverse problems with model misspecification
- Prediction scoring of data-driven discoveries for reproducible research
- Item response and response time model for personality assessment via linear ballistic accumulation
- Detecting multiple random changepoints in Bayesian piecewise growth mixture models
- Bayesian inference and dynamic prediction for multivariate longitudinal and survival data
- Recent advances in algebraic geometry and Bayesian statistics
- Bayesian compositional generalized linear models for analyzing microbiome data
- Bayesian predictive model averaging approach to joint longitudinal-survival modeling: application to an immuno-oncology clinical trial
- Approximating cross-validatory predictive evaluation in Bayesian latent variable models with integrated IS and WAIC
- Approximate leave-future-out cross-validation for Bayesian time series models
- A Bayesian approach to model individual differences and to partition individuals: case studies in growth and learning curves
- Finite-dimensional Discrete Random Structures and Bayesian Clustering
- Bayesian Multivariate Distributional Regression With Skewed Responses and Skewed Random Effects
This page was built for publication: Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q59366)
