-variational inference with statistical guarantees

DOI10.1214/19-AOS1827zbMATH Open1450.62031arXiv1710.03266OpenAlexW3029292097MaRDI QIDQ2196198FDOQ2196198

Authors: Yun Yang, Debdeep Pati, Anirban Bhattacharya 

Publication date: 28 August 2020

Published in: The Annals of Statistics (Search for Journal in Brave)

Abstract: We propose a family of variational approximations to Bayesian posterior distributions, called $alpha$-VB, with provable statistical guarantees. The standard variational approximation is a special case of $alpha$-VB with $alpha=1$. When $alphain(0,1]$, a novel class of variational inequalities are developed for linking the Bayes risk under the variational approximation to the objective function in the variational optimization problem, implying that maximizing the evidence lower bound in variational inference has the effect of minimizing the Bayes risk within the variational density family. Operating in a frequentist setup, the variational inequalities imply that point estimates constructed from the $alpha$-VB procedure converge at an optimal rate to the true parameter in a wide range of problems. We illustrate our general theory with a number of examples, including the mean-field variational approximation to (low)-high-dimensional Bayesian linear regression with spike and slab priors, mixture of Gaussian models, latent Dirichlet allocation, and (mixture of) Gaussian variational approximation in regular parametric models.

Full work available at URL: https://arxiv.org/abs/1710.03266

Recommendations

• Concentration of tempered posteriors and of their variational approximations
• Consistency of variational Bayes inference for estimation and model selection in mixtures
• Convergence rates of variational posterior distributions
• Frequentist consistency of variational Bayes
• On the properties of variational approximations of Gibbs posteriors

zbMATH Keywords

latent variable modelsvariational inferenceBayes riskevidence lower boundRényi divergence

Mathematics Subject Classification ID

Density estimation (62G07) Asymptotic properties of nonparametric inference (62G20) Interacting random processes; statistical mechanics type models; percolation theory (60K35)

Cites Work

• Graphical models, exponential families, and variational inference
• Asymptotic Statistics
• Simultaneous analysis of Lasso and Dantzig selector
• Title not available (Why is that?)
• Pattern recognition and machine learning.
• Bayesian fractional posteriors
• Title not available (Why is that?)
• Sampling-Based Approaches to Calculating Marginal Densities
• Monte Carlo sampling methods using Markov chains and their applications
• Convergence rates of posterior distributions for non iid observations
• Convergence rates of posterior distributions.
• 10.1162/jmlr.2003.3.4-5.993
• Scalable variational inference for Bayesian variable selection in regression, and its accuracy in genetic association studies
• A General Framework for Updating Belief Distributions
• Convergence properties of a general algorithm for calculating variational Bayesian estimates for a normal mixture model
• Gibbs posterior for variable selection in high-dimensional classification and data mining
• Title not available (Why is that?)
• An introduction to variational methods for graphical models
• Rényi Divergence and Kullback-Leibler Divergence
• On variational Bayes estimation and variational information criteria for linear regression models
• On the properties of variational approximations of Gibbs posteriors
• Variational Bayesian inference with Gaussian-mixture approximations
• A variational Bayes approach to variable selection
• Approximate Bayesian inference for simple mixtures

Cited In (25)

• Conditionally structured variational Gaussian approximation with importance weights
• Variational Bayes in State Space Models: Inferential and Predictive Accuracy
• Convergence rates of variational posterior distributions
• Variational Inference for Latent Space Models for Dynamic Networks
• Tightening bounds for variational inference by revisiting perturbation theory
• Variational Bayes for High-Dimensional Linear Regression With Sparse Priors
• Title not available (Why is that?)
• Concentration of tempered posteriors and of their variational approximations
• Covariances, robustness, and variational Bayes
• Bayesian brains and the Rényi divergence
• Title not available (Why is that?)
• Uncertainty quantification for sparse spectral variational approximations in Gaussian process regression
• Amortized Variational Inference: A Systematic Review
• Comprehensive study of variational Bayes classification for dense deep neural networks
• Statistical foundation of variational Bayes neural networks
• Heavy-tailed Bayesian nonparametric adaptation
• Adaptive variational Bayes: optimality, computation and applications
• Local convexity of the TAP free energy and AMP convergence for \(\mathbb{Z}_2\)-synchronization
• The computational asymptotics of Gaussian variational inference and the Laplace approximation
• Optimizing Variational Representations of Divergences and Accelerating Their Statistical Estimation
• On Negative Transfer and Structure of Latent Functions in Multioutput Gaussian Processes
• Emerging directions in Bayesian computation
• Spike and slab Bayesian sparse principal component analysis
• Alpha-divergence minimization for deep Gaussian processes
• User-friendly Introduction to PAC-Bayes Bounds

Uses Software

• Adam
• PRMLT

This page was built for publication: \(\alpha\)-variational inference with statistical guarantees

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2196198)