Approximating cross-validatory predictive evaluation in Bayesian latent variable models with integrated IS and WAIC

DOI10.1007/S11222-015-9577-2zbMATH Open1505.62248arXiv1404.2918OpenAlexW2112995480MaRDI QIDQ2628889FDOQ2628889

Authors: Longhai Li, Shi Qiu, Bei Zhang, Cindy Feng 

Publication date: 19 July 2016

Published in: Statistics and Computing (Search for Journal in Brave)

Abstract: A natural method for approximating out-of-sample predictive evaluation is leave-one-out cross-validation (LOOCV) --- we alternately hold out each case from a full data set and then train a Bayesian model using Markov chain Monte Carlo (MCMC) without the held-out; at last we evaluate the posterior predictive distribution of all cases with their actual observations. However, actual LOOCV is time-consuming. This paper introduces two methods, namely iIS and iWAIC, for approximating LOOCV with only Markov chain samples simulated from a posterior based on a extit{full} data set. iIS and iWAIC aim at improving the approximations given by importance sampling (IS) and WAIC in Bayesian models with possibly correlated latent variables. In iIS and iWAIC, we first integrate the predictive density over the distribution of the latent variables associated with the held-out without reference to its observation, then apply IS and WAIC approximations to the integrated predictive density. We compare iIS and iWAIC with other approximation methods in three real data examples that respectively use mixture models, models with correlated spatial effects, and a random effect logistic model. Our empirical results show that iIS and iWAIC give substantially better approximates than non-integrated IS and WAIC and other methods.

Full work available at URL: https://arxiv.org/abs/1404.2918

Recommendations

• scientific article
• Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC
• Efficient leave-one-out cross-validation for Bayesian non-factorized normal and Student-\(t\) models
• Conditional vs marginal estimation of the predictive loss of hierarchical models using WAIC and cross-validation
• Bayesian Model Assessment and Comparison Using Cross-Validation Predictive Densities

zbMATH Keywords

MCMCcross-validationDICWAICposterior predictive checkBayesian latent variable modelspredictive model assessment

Mathematics Subject Classification ID

Computational methods for problems pertaining to statistics (62-08) Bayesian inference (62F15) Numerical analysis or methods applied to Markov chains (65C40) Bayesian problems; characterization of Bayes procedures (62C10)

Cites Work

• A survey of Bayesian predictive methods for model assessment, selection and comparison
• Case-deletion importance sampling estimators: central limit theorems and related results
• Understanding predictive information criteria for Bayesian models
• Asymptotic equivalence of Bayes cross validation and widely applicable information criterion in singular learning theory
• Penalized loss functions for Bayesian model comparison
• On the Variability of Case-Deletion Importance Sampling Weights in the Bayesian Linear Model
• Bayesian Measures of Model Complexity and Fit
• Bayesian Model Assessment and Comparison Using Cross-Validation Predictive Densities
• Deviance information criteria for missing data models
• Marginal Likelihood from the Gibbs Output
• A STATISTICAL PARADOX
• Bayes Factors
• GPstuff: Bayesian modeling with Gaussian processes
• Monte Carlo strategies in scientific computing
• The Intrinsic Bayes Factor for Model Selection and Prediction
• Title not available (Why is that?)
• Bayesian Inference in Econometric Models Using Monte Carlo Integration
• Density Estimation With Confidence Sets Exemplified by Superclusters and Voids in the Galaxies
• Bayesian predictive information criterion for the evaluation of hierarchical Bayesian and empirical Bayes models
• Simulating normalizing constants: From importance sampling to bridge sampling to path sampling
• Algebraic Geometry and Statistical Learning Theory
• Hierarchical Spatio-Temporal Mapping of Disease Rates
• Title not available (Why is that?)
• Equations of states in singular statistical estimation
• Bayesian Hypothesis Testing: A Reference Approach
• Bayesian learning for neural networks
• Bayesian hypothesis testing in latent variable models
• A new approach to Bayesian hypothesis testing
• Difficulty of selecting among multilevel models using predictive accuracy
• Bayesian analysis of stochastic volatility models
• Identifying outliers in Bayesian hierarchical models: a simulation-based approach
• Stochastic volatility modelling in continuous time with general marginal distributions: inference, prediction and model selection
• Properties of intrinsic and fractional Bayes factors
• Importance re-sampling MCMC for cross-validation in inverse problems
• Deviance information criterion for latent variable models and misspecified models

Cited In (8)

• Prediction scoring of data-driven discoveries for reproducible research
• Efficient leave-one-out cross-validation for Bayesian non-factorized normal and Student-\(t\) models
• Bayesian model selection in linear mixed models for longitudinal data
• Model selection for Bayesian linear mixed models with longitudinal data: Sensitivity to the choice of priors
• Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC
• Bayesian model selection for multilevel mediation models
• Conditional vs marginal estimation of the predictive loss of hierarchical models using WAIC and cross-validation
• Bayesian comparison of latent variable models: conditional versus marginal likelihoods

Uses Software

• JAGS
• GPstuff

This page was built for publication: Approximating cross-validatory predictive evaluation in Bayesian latent variable models with integrated IS and WAIC

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2628889)