-variational inference with statistical guarantees
From MaRDI portal
Publication:2196198
Abstract: We propose a family of variational approximations to Bayesian posterior distributions, called -VB, with provable statistical guarantees. The standard variational approximation is a special case of -VB with . When , a novel class of variational inequalities are developed for linking the Bayes risk under the variational approximation to the objective function in the variational optimization problem, implying that maximizing the evidence lower bound in variational inference has the effect of minimizing the Bayes risk within the variational density family. Operating in a frequentist setup, the variational inequalities imply that point estimates constructed from the -VB procedure converge at an optimal rate to the true parameter in a wide range of problems. We illustrate our general theory with a number of examples, including the mean-field variational approximation to (low)-high-dimensional Bayesian linear regression with spike and slab priors, mixture of Gaussian models, latent Dirichlet allocation, and (mixture of) Gaussian variational approximation in regular parametric models.
The authors research issues on point estimation where data are coming from ``mean-field or other variational approximations to a Bayesian posterior. They derive that there is no reason for denying or preferring it with respect to using the true posterior in terms of accuracy. The affirmation seems to be true for a broad class of statistical models, particularly for the class of variational objective functions indexed by a temperature parameter. For the particular case of $\alpha = 1$ bounds for the risk characterize the minimax optimality of variational of the point estimates. The authors analyze the case $\alpha < 1$. Dealing with a prior mass condition supports the derivation of the risk bounds of Rényi's divergence. They consider some particular problems for illustrating the behavior of their proposal, and claim that under the fulfillment of all the hypotheses the rates of convergence are the same whenever $\alpha\in (0,1]$.
Recommendations
- Concentration of tempered posteriors and of their variational approximations
- Consistency of variational Bayes inference for estimation and model selection in mixtures
- Convergence rates of variational posterior distributions
- Frequentist consistency of variational Bayes
- On the properties of variational approximations of Gibbs posteriors
Cites work
- 10.1162/jmlr.2003.3.4-5.993
- A General Framework for Updating Belief Distributions
- A variational Bayes approach to variable selection
- An introduction to variational methods for graphical models
- Approximate Bayesian inference for simple mixtures
- Asymptotic Statistics
- Bayesian fractional posteriors
- Convergence properties of a general algorithm for calculating variational Bayesian estimates for a normal mixture model
- Convergence rates of posterior distributions for non iid observations
- Convergence rates of posterior distributions.
- Gibbs posterior for variable selection in high-dimensional classification and data mining
- Graphical models, exponential families, and variational inference
- scientific article; zbMATH DE number 6377992 (Why is no real title available?)
- scientific article; zbMATH DE number 1043533 (Why is no real title available?)
- scientific article; zbMATH DE number 2117879 (Why is no real title available?)
- Monte Carlo sampling methods using Markov chains and their applications
- On the properties of variational approximations of Gibbs posteriors
- On variational Bayes estimation and variational information criteria for linear regression models
- Pattern recognition and machine learning.
- Rényi Divergence and Kullback-Leibler Divergence
- Sampling-Based Approaches to Calculating Marginal Densities
- Scalable variational inference for Bayesian variable selection in regression, and its accuracy in genetic association studies
- Simultaneous analysis of Lasso and Dantzig selector
- Variational Bayesian inference with Gaussian-mixture approximations
Cited in
(42)- Alpha-divergence minimization for deep Gaussian processes
- The computational asymptotics of Gaussian variational inference and the Laplace approximation
- Concentration of tempered posteriors and of their variational approximations
- Conditionally structured variational Gaussian approximation with importance weights
- Convergence rates of variational posterior distributions
- Covariances, robustness, and variational Bayes
- When random initializations help: a study of variational inference for community detection
- Variational Inference for Latent Space Models for Dynamic Networks
- Bayesian brains and the Rényi divergence
- Optimizing Variational Representations of Divergences and Accelerating Their Statistical Estimation
- scientific article; zbMATH DE number 7306861 (Why is no real title available?)
- Tightening bounds for variational inference by revisiting perturbation theory
- Variational Bayes for High-Dimensional Linear Regression With Sparse Priors
- Comprehensive study of variational Bayes classification for dense deep neural networks
- Statistical foundation of variational Bayes neural networks
- On Negative Transfer and Structure of Latent Functions in Multioutput Gaussian Processes
- User-friendly Introduction to PAC-Bayes Bounds
- Variational Bayes in State Space Models: Inferential and Predictive Accuracy
- Local convexity of the TAP free energy and AMP convergence for \(\mathbb{Z}_2\)-synchronization
- Uncertainty quantification for sparse spectral variational approximations in Gaussian process regression
- Adaptive variational Bayes: optimality, computation and applications
- Amortized Variational Inference: A Systematic Review
- Emerging directions in Bayesian computation
- Spike and slab Bayesian sparse principal component analysis
- Heavy-tailed Bayesian nonparametric adaptation
- Adaptive posterior concentration rates for sparse high-dimensional linear regression with random design and unknown error variance
- Posterior and variational inference for deep neural networks with heavy-tailed weights
- Federated causal inference from observational data
- On properties of fractional posterior in generalized reduced-rank regression
- Projection-posterior for variable selection: weak limit and coverage
- Predictive performance of power posteriors
- On the convergence of coordinate ascent variational inference
- Variational inference for the latent shrinkage position model
- Concentration of a sparse Bayesian model with horseshoe prior in estimating high-dimensional precision matrix
- Loss-Based Variational Bayes Prediction
- Concentration properties of fractional posterior in 1-bit matrix completion
- Variational Inference Aided Variable Selection For Spatially Structured High Dimensional Covariates
- Evidence bounds in singular models: probabilistic and variational perspectives
- Consistency of variational inference for Besov priors in non-linear inverse problems
- Bayesian inference in high-dimensional models
- Emerging Knowledge Trend in Statistical Research: A Content-Based Analysis Using Covariate-Assisted Dynamic Topic Model
- Fluctuations in random field Ising models
This page was built for publication: \(\alpha\)-variational inference with statistical guarantees
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2196198)
