-variational inference with statistical guarantees

DOI10.1214/19-AOS1827MaRDI QIDQ2196198zbMATH OpenOpenAlexFDO

Authors Yun Yang, Debdeep Pati, Anirban Bhattacharya

Publication date 28 August 2020

Published in The Annals of Statistics (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1710.03266, https://projecteuclid.org/euclid.aos/1590480038

zbMATH Keywords

latent variable modelsvariational inferenceBayes riskevidence lower boundRényi divergence

Mathematics Subject Classification ID

Density estimation (62G07) Asymptotic properties of nonparametric inference (62G20) Interacting random processes; statistical mechanics type models; percolation theory (60K35)

Abstract: We propose a family of variational approximations to Bayesian posterior distributions, called $alpha$-VB, with provable statistical guarantees. The standard variational approximation is a special case of $alpha$-VB with $alpha=1$. When $alphain(0,1]$, a novel class of variational inequalities are developed for linking the Bayes risk under the variational approximation to the objective function in the variational optimization problem, implying that maximizing the evidence lower bound in variational inference has the effect of minimizing the Bayes risk within the variational density family. Operating in a frequentist setup, the variational inequalities imply that point estimates constructed from the $alpha$-VB procedure converge at an optimal rate to the true parameter in a wide range of problems. We illustrate our general theory with a number of examples, including the mean-field variational approximation to (low)-high-dimensional Bayesian linear regression with spike and slab priors, mixture of Gaussian models, latent Dirichlet allocation, and (mixture of) Gaussian variational approximation in regular parametric models.

Recommendations

• Concentration of tempered posteriors and of their variational approximations
• Consistency of variational Bayes inference for estimation and model selection in mixtures
• Convergence rates of variational posterior distributions
• Frequentist consistency of variational Bayes
• On the properties of variational approximations of Gibbs posteriors

Cites work

• scientific article; zbMATH DE number 6377992 (Why is no real title available?)
• scientific article; zbMATH DE number 1043533 (Why is no real title available?)
• scientific article; zbMATH DE number 2117879 (Why is no real title available?)
• 10.1162/jmlr.2003.3.4-5.993
• A General Framework for Updating Belief Distributions
• A variational Bayes approach to variable selection
• An introduction to variational methods for graphical models
• Approximate Bayesian inference for simple mixtures
• Asymptotic Statistics
• Bayesian fractional posteriors
• Convergence properties of a general algorithm for calculating variational Bayesian estimates for a normal mixture model
• Convergence rates of posterior distributions for non iid observations
• Convergence rates of posterior distributions.
• Gibbs posterior for variable selection in high-dimensional classification and data mining
• Graphical models, exponential families, and variational inference
• Monte Carlo sampling methods using Markov chains and their applications
• On the properties of variational approximations of Gibbs posteriors
• On variational Bayes estimation and variational information criteria for linear regression models
• Pattern recognition and machine learning.
• Rényi Divergence and Kullback-Leibler Divergence
• Sampling-Based Approaches to Calculating Marginal Densities
• Scalable variational inference for Bayesian variable selection in regression, and its accuracy in genetic association studies
• Simultaneous analysis of Lasso and Dantzig selector
• Variational Bayesian inference with Gaussian-mixture approximations

Cited in

(25)

• Conditionally structured variational Gaussian approximation with importance weights
• User-friendly Introduction to PAC-Bayes Bounds
• Variational Bayes in State Space Models: Inferential and Predictive Accuracy
• Convergence rates of variational posterior distributions
• Variational Inference for Latent Space Models for Dynamic Networks
• Tightening bounds for variational inference by revisiting perturbation theory
• Variational Bayes for High-Dimensional Linear Regression With Sparse Priors
• scientific article; zbMATH DE number 7306861 (Why is no real title available?)
• Concentration of tempered posteriors and of their variational approximations
• Covariances, robustness, and variational Bayes
• Bayesian brains and the Rényi divergence
• scientific article; zbMATH DE number 7370539 (Why is no real title available?)
• Uncertainty quantification for sparse spectral variational approximations in Gaussian process regression
• Amortized Variational Inference: A Systematic Review
• Comprehensive study of variational Bayes classification for dense deep neural networks
• Statistical foundation of variational Bayes neural networks
• Heavy-tailed Bayesian nonparametric adaptation
• Adaptive variational Bayes: optimality, computation and applications
• Local convexity of the TAP free energy and AMP convergence for \(\mathbb{Z}_2\)-synchronization
• The computational asymptotics of Gaussian variational inference and the Laplace approximation
• Optimizing Variational Representations of Divergences and Accelerating Their Statistical Estimation
• On Negative Transfer and Structure of Latent Functions in Multioutput Gaussian Processes
• Emerging directions in Bayesian computation
• Spike and slab Bayesian sparse principal component analysis
• Alpha-divergence minimization for deep Gaussian processes

Describes a project that uses

Uses Software

• PRMLT
• Adam

This page was built for publication: \(\alpha\)-variational inference with statistical guarantees

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2196198)