Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC
From MaRDI portal
Abstract: Leave-one-out cross-validation (LOO) and the widely applicable information criterion (WAIC) are methods for estimating pointwise out-of-sample prediction accuracy from a fitted Bayesian model using the log-likelihood evaluated at the posterior simulations of the parameter values. LOO and WAIC have various advantages over simpler estimates of predictive error such as AIC and DIC but are less used in practice because they involve additional computational steps. Here we lay out fast and stable computations for LOO and WAIC that can be performed using existing simulation draws. We introduce an efficient computation of LOO using Pareto-smoothed importance sampling (PSIS), a new procedure for regularizing importance weights. Although WAIC is asymptotically equal to LOO, we demonstrate that PSIS-LOO is more robust in the finite case with weak priors or influential observations. As a byproduct of our calculations, we also obtain approximate standard errors for estimated predictive errors and for comparing of predictive errors between two models. We implement the computations in an R package called 'loo' and demonstrate using models fit with the Bayesian inference package Stan.
Recommendations
- Efficient leave-one-out cross-validation for Bayesian non-factorized normal and Student-t models
- scientific article; zbMATH DE number 6617268
- Approximating cross-validatory predictive evaluation in Bayesian latent variable models with integrated IS and WAIC
- Bayesian Model Assessment and Comparison Using Cross-Validation Predictive Densities
- Bayesian Measures of Model Complexity and Fit
Cites work
- A comparative study of ordinary cross-validation, v-fold cross-validation and the repeated learning-testing methods
- A Predictive Approach to Model Selection
- A survey of Bayesian predictive methods for model assessment, selection and comparison
- A survey of cross-validation procedures for model selection
- Asymptotic equivalence of Bayes cross validation and widely applicable information criterion in singular learning theory
- Bayesian data analysis.
- Bayesian Measures of Model Complexity and Fit
- Bayesian Model Assessment and Comparison Using Cross-Validation Predictive Densities
- Bayesian model averaging: A tutorial. (with comments and a rejoinder).
- Case-deletion importance sampling estimators: central limit theorems and related results
- Comparison of Bayesian predictive methods for model selection
- DIC in variable selection
- Erratum to: ``Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC
- scientific article; zbMATH DE number 6617268 (Why is no real title available?)
- scientific article; zbMATH DE number 3553528 (Why is no real title available?)
- scientific article; zbMATH DE number 578421 (Why is no real title available?)
- scientific article; zbMATH DE number 3444596 (Why is no real title available?)
- scientific article; zbMATH DE number 849928 (Why is no real title available?)
- Laplace approximation for logistic Gaussian process density estimation and regression
- On the Variability of Case-Deletion Importance Sampling Weights in the Bayesian Linear Model
- Penalized loss functions for Bayesian model comparison
- Strictly Proper Scoring Rules, Prediction, and Estimation
- Testing the assumptions behind importance sampling
- The no-U-turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo
- Understanding predictive information criteria for Bayesian models
Cited in
(only showing first 100 items - show all)- TriDimRegression
- Using stacking to average Bayesian predictive distributions (with discussion)
- Bayesian inference for conditional copulas using Gaussian process single index models
- Optimal group decision: a matter of confidence calibration
- Conditional vs marginal estimation of the predictive loss of hierarchical models using WAIC and cross-validation
- The quantile probability model
- Bayesian model selection in the \(\mathcal{M}\)-open setting -- approximate posterior inference and subsampling for efficient large-scale leave-one-out cross-validation via the difference estimator
- Stochastic optimization with adaptive restart: a framework for integrated local and global learning
- An extended two-stage sequential optimization approach: properties and performance
- Efficient leave-one-out cross-validation for Bayesian non-factorized normal and Student-t models
- A Bayesian model of microbiome data for simultaneous identification of covariate associations and prediction of phenotypic outcomes
- Implicitly adaptive importance sampling
- A spatial mixed-effects regression model for electoral data
- Bayesian spatial modeling for housing data in South Africa
- Transfer of macroeconomic shocks in stress tests modeling
- A Bayesian generalized explanatory item response model to account for learning during the test
- A Bayesian time-varying effect model for behavioral mHealth data
- Region-referenced spectral power dynamics of EEG signals: a hierarchical modeling approach
- Estimating the stillbirth rate for 195 countries using a Bayesian sparse regression model with temporal smoothing
- Bayesian multivariate sparse functional principal components analysis with application to longitudinal microbiome multiomics data
- On outliers detection and prior distribution sensitivity in standard skew-probit regression models
- Joint modeling of distances and times in point-count surveys
- Modeling obesity rate with spatial auto-correlation: a case study
- Bayesian mixture model of extended redundancy analysis
- The reliability factor: modeling individual reliability with multiple items from a single assessment
- Prediction scoring of data-driven discoveries for reproducible research
- A Bayesian approach to model individual differences and to partition individuals: case studies in growth and learning curves
- Fundamental tools for developing likelihood functions within ACT-R
- Support provided by elderly in Italy: a hierarchical analysis of ego networks controlling for alter-overlapping
- Detecting and modeling changes in a time series of proportions
- Bayesian mitigation of spatial coarsening for a Hawkes model applied to gunfire, wildfire and viral contagion
- Bayesian forecasting with a regime-switching zero-inflated multilevel Poisson regression model: an application to adolescent alcohol use with spatial covariates
- Software defect prediction model based on distance metric learning
- Bayesian functional registration of fMRI activation maps
- Bayesian mixed effects models for zero-inflated compositions in microbiome data analysis
- Projective inference in high-dimensional problems: prediction and feature selection
- Extending RT-MPTs to enable equal process times
- Modeling COVID-19 pandemic using Bayesian analysis with application to Slovene data
- Development of a novel computational model for the balloon analogue risk task: the exponential-weight mean-variance model
- Inflection points in community-level homeless rates
- Bayesian cylindrical data modeling using Abe-Ley mixtures
- Detecting multiple random changepoints in Bayesian piecewise growth mixture models
- Bayesian model-based clustering for longitudinal ordinal data
- Item response and response time model for personality assessment via linear ballistic accumulation
- A simple method for comparing complex models: Bayesian model comparison for hierarchical multinomial processing tree models using Warp-III bridge sampling
- Bayesian comparison of latent variable models: conditional versus marginal likelihoods
- A hierarchical Bayesian state trace analysis for assessing monotonicity while factoring out subject, item, and trial level dependencies
- Investigation of the widely applicable Bayesian information criterion
- Bayesian inference and model comparison for metallic fatigue data
- Higher order equivalence of Bayes cross validation and WAIC
- Approximating cross-validatory predictive evaluation in Bayesian latent variable models with integrated IS and WAIC
- Neuro-cognitive models of single-trial EEG measures describe latent effects of spatial attention during perceptual decision making
- History and nature of the Jeffreys-Lindley paradox
- Bayesian clustering of spatial functional data with application to a human mobility study during COVID-19
- The leave-worst-k-out criterion for cross validation
- loo
- scientific article; zbMATH DE number 6617268 (Why is no real title available?)
- Bayesian residual analysis for spatially correlated data
- Bayesian Multivariate Distributional Regression With Skewed Responses and Skewed Random Effects
- Bayesian Model Assessment and Comparison Using Cross-Validation Predictive Densities
- Bayesian structured antedependence model proposals for longitudinal data
- Bayesian estimation of a flexible bifactor generalized partial credit model to survey data
- Approximate leave-future-out cross-validation for Bayesian time series models
- Quantifying conditional probability tables in Bayesian networks: Bayesian regression for scenario-based encoding of elicited expert assessments on feral pig habitat
- A Bayesian Approach to Envelope Quantile Regression
- Semiparametric regression for dual population mortality
- Massive Parallelization Boosts Big Bayesian Multidimensional Scaling
- Bayesian Variable Selection for Gaussian Copula Regression Models
- A flexible procedure for formulating probability distributions on the unit interval with applications
- Bayesian Approaches to Shrinkage and Sparse Estimation
- Performance of asymmetric links and correction methods for imbalanced data in binary regression
- Bayesian dynamic modelling to assess differential treatment effects on panic attack frequencies
- Probabilistic approach to limited-data computed tomography reconstruction
- Analysis of single-index models with scale mixture of normals errors by using Bayesian P-splines
- PARSIMONIOUS PARAMETERIZATION OF AGE-PERIOD-COHORT MODELS BY BAYESIAN SHRINKAGE
- Bayesian hierarchical models for the prediction of volleyball results
- R-squared for Bayesian Regression Models
- Marginal Likelihood Computation for Model Selection and Hypothesis Testing: An Extensive Review
- Model-based clustering of trends and cycles of nitrate concentrations in rivers across France
- Generalized Bayes approach to inverse problems with model misspecification
- A Bayesian spatial model for imaging genetics
- The Dirichlet dual response model: an item response model for continuous bounded interval responses
- Regression modelling with the tilted beta distribution: A Bayesian approach
- Disease mapping method comparing the spatial distribution of a disease with a control disease
- A Richards growth model to predict fruit weight
- Bayesian nonparametric quantile process regression and estimation of marginal quantile effects
- Multivariate Conway-Maxwell-Poisson Distribution: Sarmanov Method and Doubly Intractable Bayesian Inference
- Bayesian analysis of first-order Markov models for autocorrelated binary responses
- How close and how much? Linking health outcomes to built environment spatial distributions
- Using reference models in variable selection
- The role of passing network indicators in modeling football outcomes: an application using Bayesian hierarchical models
- Hierarchical Bayes modelling of penalty conversion rates of bundesliga players
- Parameter continuity in time-varying Gauss-Markov models for learning from small training data sets
- An explanatory mixture <scp>IRT</scp> model for careless and insufficient effort responding in self‐report measures
- Exploring examinees' responses to constructed response items with a supervised topic model
- Extending exploratory diagnostic classification models: Inferring the effect of covariates
- Longitudinal binary response models using alternative links for medical data
- Bayesian inference and dynamic prediction for multivariate longitudinal and survival data
- Recent advances in algebraic geometry and Bayesian statistics
- Bayesian survival analysis of Rayleigh-X family with time varying covariate
This page was built for publication: Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q59366)
