Expectation Propagation in the Large Data Limit
From MaRDI portal
Abstract: Expectation Propagation (Minka, 2001) is a widely successful algorithm for variational inference. EP is an iterative algorithm used to approximate complicated distributions, typically to find a Gaussian approximation of posterior distributions. In many applications of this type, EP performs extremely well. Surprisingly, despite its widespread use, there are very few theoretical guarantees on Gaussian EP, and it is quite poorly understood. In order to analyze EP, we first introduce a variant of EP: averaged-EP (aEP), which operates on a smaller parameter space. We then consider aEP and EP in the limit of infinite data, where the overall contribution of each likelihood term is small and where posteriors are almost Gaussian. In this limit, we prove that the iterations of both aEP and EP are simple: they behave like iterations of Newton's algorithm for finding the mode of a function. We use this limit behavior to prove that EP is asymptotically exact, and to obtain other insights into the dynamic behavior of EP: for example, that it may diverge under poor initialization exactly like Newton's method. EP is a simple algorithm to state, but a difficult one to study. Our results should facilitate further research into the theoretical properties of this important method.
Recommendations
- Expectation propagation
- Expectation propagation for likelihood-free inference
- Expectation propagation for continuous time stochastic processes
- On expectation propagation for generalised, linear and mixed models
- Expectation propagation for Poisson data
- Expectation propagation as a way of life: a framework for Bayesian inference on partitioned data
- Large dimensional empirical likelihood
- Empirical Bayes estimates for large-scale prediction problems
- Infinite Dirichlet mixture models learning via expectation propagation
- Expectation propagation with factorizing distributions: a Gaussian approximation and performance results for simple models
Cites work
- Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations (with discussion)
- Approximations for binary Gaussian process classification
- Assessing approximate inference for binary Gaussian process classification.
- Best constants in Young's inequality, its converse, and its generalization to more than three functions
- Expectation consistent approximate inference
- Expectation propagation for likelihood-free inference
- Expectation propagation for nonlinear inverse problems -- with an application to electrical impedance tomography
- Expectation propagation with factorizing distributions: a Gaussian approximation and performance results for simple models
- scientific article; zbMATH DE number 6378124 (Why is no real title available?)
- Log-concavity and strong log-concavity: a review
- Perturbation corrections in approximate inference: mixture modelling applications
- Reinforcement of an inequality due to Brascamp and Lieb
- Robust Gaussian process regression with a Student-t likelihood
- The Bernstein-von Mises theorem under misspecification
Cited in
(24)- Leave Pima Indians alone: binary regression as a benchmark for Bayesian computation
- Infinite-dimensional gradient-based descent for alpha-divergence minimisation
- Factor graph fragmentization of expectation propagation
- The explicit form of expectation propagation for a simple statistical model
- Perturbative corrections for approximate inference in Gaussian latent variable models
- scientific article; zbMATH DE number 6378124 (Why is no real title available?)
- Expectation propagation for continuous time stochastic processes
- Expectation propagation
- Expectation propagation with factorizing distributions: a Gaussian approximation and performance results for simple models
- Distributed Bayesian learning with stochastic natural gradient expectation propagation and the posterior server
- On expectation propagation for generalised, linear and mixed models
- Expectation propagation as a way of life: a framework for Bayesian inference on partitioned data
- Expectation propagation for likelihood-free inference
- Analysis of Bayesian inference algorithms by the dynamical functional approach
- Bayesian Approaches to Shrinkage and Sparse Estimation
- Fast and accurate binary response mixed model analysis via expectation propagation
- Maximum likelihood for Gaussian process classification and generalized linear mixed models under case-control sampling
- Expectation propagation for Poisson data
- Bayesian Conjugacy in Probit, Tobit, Multinomial Probit and Extensions: A Review and New Results
- Gaussian process regression in the flat limit
- On the approximation accuracy of Gaussian variational inference
- A Bayesian practitioner's guide to expectation propagation
- Multivariate Gaussian cumulative distribution functions as the marginal likelihood of their dual Bayesian probit models
- Skewed Bernstein-von Mises theorem and skew-modal approximations
This page was built for publication: Expectation Propagation in the Large Data Limit
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q4603808)
