Dynamics of Bayesian updating with dependent data and misspecified models

DOI10.1214/09-EJS485zbMATH Open1326.62017arXiv0901.1342MaRDI QIDQ1952015FDOQ1952015

Cosma Rohilla Shalizi

Publication date: 27 May 2013

Published in: Electronic Journal of Statistics (Search for Journal in Brave)

Abstract: Much is now known about the consistency of Bayesian updating on infinite-dimensional parameter spaces with independent or Markovian data. Necessary conditions for consistency include the prior putting enough weight on the correct neighborhoods of the data-generating distribution; various sufficient conditions further restrict the prior in ways analogous to capacity control in frequentist nonparametrics. The asymptotics of Bayesian updating with mis-specified models or priors, or non-Markovian data, are far less well explored. Here I establish sufficient conditions for posterior convergence when all hypotheses are wrong, and the data have complex dependencies. The main dynamical assumption is the asymptotic equipartition (Shannon-McMillan-Breiman) property of information theory. This, along with Egorov's Theorem on uniform convergence, lets me build a sieve-like structure for the prior. The main statistical assumption, also a form of capacity control, concerns the compatibility of the prior and the data-generating process, controlling the fluctuations in the log-likelihood when averaged over the sieve-like sets. In addition to posterior convergence, I derive a kind of large deviations principle for the posterior measure, extending in some cases to rates of convergence, and discuss the advantages of predicting using a combination of models known to be wrong. An appendix sketches connections between these results and the replicator dynamics of evolutionary theory.

Full work available at URL: https://arxiv.org/abs/0901.1342

zbMATH Keywords

Bayesian nonparametricslarge deviationsreplicator dynamicsposterior convergenceBayesian consistencysofic systemsEgorov's theoremasymptotic equipartition

Mathematics Subject Classification ID

Asymptotic properties of nonparametric inference (62G20) Markov processes: estimation; hidden Markov models (62M05) Large deviations (60F10) Non-Markovian processes: estimation (62M09) Problems related to evolution (92D15) Bayesian problems; characterization of Bayes procedures (62C10) Measures of information, entropy (94A17)

Cites Work

• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Likelihood Ratio Tests for Model Selection and Non-Nested Hypotheses
• Prediction, Learning, and Games
• Bayesian Estimation of the Spectral Density of a Time Series
• Convergence rates of posterior distributions for non iid observations
• On the consistency of Bayes estimates
• Convergence rates of posterior distributions.
• A sandwich proof of the Shannon-McMillan-Breiman theorem
• Misspecification in infinite-dimensional Bayesian statistics
• Limiting Behavior of Posterior Distributions when the Model is Incorrect
• Subshifts of finite type and sofic systems
• Merging of Opinions with Increasing Information
• Evolutionary Games and Population Dynamics
• An Introduction to Symbolic Dynamics and Coding
• Symbolic dynamics. One-sided, two-sided and countable state Markov shifts
• Rates of convergence of posterior distributions.
• Bayesian nonparametrics
• New approaches to Bayesian consistency
• Theory of statistics
• The consistency of posterior distributions in nonparametric problems
• On Bayes procedures
• The multiplicative weights update method: a meta-algorithm and applications
• Model selection tests for nonlinear dynamic models
• Consistency a Posteriori
• Hidden Markov Models and Dynamical Systems
• Complexity
• Computational mechanics: pattern and prediction, structure and simplicity.
• Nonparametric maximum likelihood estimation by the method of sieves
• Sufficient statistics and extreme points
• How sampling reveals a process
• Learning through reinforcement and replicator dynamics
• Nonparametric time series prediction through adaptive model selection
• Moderate Deviations for Bayes Posteriors
• Some PAC-Bayesian theorems
• From \(\varepsilon\)-entropy to KL-entropy: analysis of minimum information complexity density estima\-tion
• Finitary measures for subshifts of finite type and sofic systems
• Correction to: Entropy and the consistent estimation of joint distributions
• From finite to infinite range order via annealing: the causal architecture of deformation faulting in annealed close-packed crystals
• Sufficient conditions for Bayesian consistency
• A predictive view of continuous time processes
• Convergence properties of sequential Bayesian \(D\)-optimal designs
• Some Sufficient Conditions on an Arbitrary Class of Stochastic Processes for the Existence of a Predictor
• Bayesian consistency for stationary models
• REINFORCEMENT LEARNING IN MARKOVIAN EVOLUTIONARY GAMES
• Large deviations and the Bayesian estimation of higher-order Markov transition functions
• On Rates of Convergence for Posterior Distributions Under Misspecification

Cited In (23)

• Title not available (Why is that?)
• Asymptotic theory of dependent Bayesian multiple testing procedures under possible model misspecification
• Non-marginal decisions: a novel Bayesian multiple testing procedure
• On clonal selection
• Decentralized pricing and the equivalence between Nash and Walrasian equilibrium
• On posterior concentration in misspecified models
• A Short Note on Almost Sure Convergence of Bayes Factors in the General Set-Up
• Independence and Bayesian updating methods
• Posterior consistency for the spectral density of non‐Gaussian stationary time series
• On the Importance of the Jacobian Determinant in Parameter Inference for Random Parameter and Random Measurement Error Models
• Bio-inspired feedback-circuit implementation of discrete, free energy optimizing, winner-take-all computations
• Adaptive Bayesian density estimation with location-scale mixtures
• On the brittleness of Bayesian inference
• Statistical inference for dynamical systems: a review
• Runs, panics and bubbles: Diamond-Dybvig and Morris-Shin reconsidered
• General Robust Bayes Pseudo-Posteriors: Exponential Convergence Results with Applications
• Experimental Design for Nonparametric Correction of Misspecified Dynamical Models
• On classical and Bayesian asymptotics in state space stochastic differential equations
• Darwinian evolution as Brownian motion on the simplex: a geometric perspective on stochastic replicator dynamics
• Optimal point process filtering and estimation of the coalescent process
• Consistency of maximum likelihood estimation for some dynamical systems
• Bayesian Model Comparison with the Hyvärinen Score: Computation and Consistency
• Gibbs posterior convergence and the thermodynamic formalism

Recommendations

• Bayesian inference with misspecified models 👍 👎
• Independence assumptions and Bayesian updating 👍 👎
• Independence and Bayesian updating methods 👍 👎
• Bayesian updating revisited 👍 👎
• Bayesian Updating with Confounded Signals 👍 👎
• Bayesian model updating with consideration of modeling error 👍 👎
• Title not available (Why is that?) 👍 👎
• Title not available (Why is that?) 👍 👎
• Title not available (Why is that?) 👍 👎
• Title not available (Why is that?) 👍 👎

This page was built for publication: Dynamics of Bayesian updating with dependent data and misspecified models

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q1952015)