Concentration of tempered posteriors and of their variational approximations
From MaRDI portal
Publication:2196229
Abstract: While Bayesian methods are extremely popular in statistics and machine learning, their application to massive datasets is often challenging, when possible at all. Indeed, the classical MCMC algorithms are prohibitively slow when both the model dimension and the sample size are large. Variational Bayesian methods aim at approximating the posterior by a distribution in a tractable family. Thus, MCMC are replaced by an optimization algorithm which is orders of magnitude faster. VB methods have been applied in such computationally demanding applications as including collaborative filtering, image and video processing, NLP and text processing... However, despite very nice results in practice, the theoretical properties of these approximations are usually not known. In this paper, we propose a general approach to prove the concentration of variational approximations of fractional posteriors. We apply our theory to two examples: matrix completion, and Gaussian VB.
The authors consider variational Bayes (VB) fast Bayesian inference to compute variational approximations of the posterior for massive datasets where sampling schemes such as Markov chain Monte Carlo are computationally impractical. Variational Bayesian methods aim at approximating the posterior by a distribution in a tractable family \(F\). The authors propose a general oracle inequality that relates the quality of the VB approximation to the prior \(\pi\) and to the structure of \(F\). They provide a simple condition that allows to derive rates of convergence from this oracle inequality. Various examples are presented. It is shown that for parametric models with log-Lipschitz likelihood, Gaussian VB leads to efficient algorithms and consistent estimators. A matrix completion is demonstrated as a high-dimensional example, and a density estimation is presented as a nonparametric example.
Recommendations
- Frequentist consistency of variational Bayes
- Convergence rates of variational posterior distributions
- \(\alpha\)-variational inference with statistical guarantees
- Consistency of variational Bayes inference for estimation and model selection in mixtures
- On the properties of variational approximations of Gibbs posteriors
Cites work
- 1-bit matrix completion: PAC-Bayesian analysis of a variational approximation
- A Bayesian approach for noisy matrix completion: optimal rate under general sampling distribution
- Active Learning and Basis Selection for Kernel-Based Linear Models: A Bayesian Perspective
- Bayesian fractional posteriors
- Convergence rates of posterior distributions.
- Estimation bounds and sharp oracle inequalities of regularized procedures with Lipschitz loss functions
- Frequentist consistency of variational Bayes
- Fundamentals of nonparametric Bayesian inference
- scientific article; zbMATH DE number 6377992 (Why is no real title available?)
- scientific article; zbMATH DE number 5544465 (Why is no real title available?)
- Inconsistency of Bayesian inference for misspecified linear models, and a proposal for repairing it
- Introduction to nonparametric estimation
- Leave Pima Indians alone: binary regression as a benchmark for Bayesian computation
- Noisy low-rank matrix completion with general sampling distribution
- Nuclear-norm penalization and optimal rates for noisy low-rank matrix completion
- On the properties of variational approximations of Gibbs posteriors
- Online learning and online convex optimization
- Pattern recognition and machine learning.
- Rényi Divergence and Kullback-Leibler Divergence
- Statistical learning theory and stochastic optimization. Ecole d'Eté de Probabilitiés de Saint-Flour XXXI -- 2001.
- The Power of Convex Relaxation: Near-Optimal Matrix Completion
- The safe Bayesian. Learning the learning rate via the mixability gap
- The Variational Gaussian Approximation Revisited
Cited in
(57)- Numerical comparisons between Bayesian and frequentist low-rank matrix completion: estimation accuracy and uncertainty quantification
- Consistency of variational Bayes inference for estimation and model selection in mixtures
- Variational Bayesian methods for spatial data analysis
- Variational Bayes on manifolds
- Updating variational Bayes: fast sequential posterior inference
- Tight risk bound for high dimensional time series completion
- The computational asymptotics of Gaussian variational inference and the Laplace approximation
- \(\alpha\)-variational inference with statistical guarantees
- Convergence rates of variational posterior distributions
- Divergence measures and a general framework for local variational approximation
- Variational inference for generalized linear mixed models using partially noncentered parametrizations
- Condition for perfect dimensionality recovery by variational Bayesian PCA
- On variational Bayes estimation and variational information criteria for linear regression models
- Approximate Riemannian conjugate gradient learning for fixed-form variational Bayes
- On the properties of variational approximations of Gibbs posteriors
- Covariances, robustness, and variational Bayes
- VAE-KRnet and its applications to variational Bayes
- Unbiased MLMC-based variational Bayes for likelihood-free inference
- Variational Bayes in private settings (VIPS)
- scientific article; zbMATH DE number 7306861 (Why is no real title available?)
- Frequentist consistency of variational Bayes
- Variational Bayesian methods: concepts and neuroimage applications
- Averaged collapsed variational Bayes inference
- Variational Bayes for High-Dimensional Linear Regression With Sparse Priors
- Variational Bayesian approximation. A rigorous approach
- Comprehensive study of variational Bayes classification for dense deep neural networks
- Statistical foundation of variational Bayes neural networks
- User-friendly Introduction to PAC-Bayes Bounds
- Projective Integral Updates for High-Dimensional Variational Inference
- Local convexity of the TAP free energy and AMP convergence for \(\mathbb{Z}_2\)-synchronization
- Uncertainty quantification for sparse spectral variational approximations in Gaussian process regression
- Online estimation and community detection of network point processes for event streams
- Adaptive variational Bayes: optimality, computation and applications
- Approximating Bayes in the 21st century
- Emerging directions in Bayesian computation
- Asymptotic consistency of loss-calibrated variational Bayes
- On the approximation accuracy of Gaussian variational inference
- Heavy-tailed Bayesian nonparametric adaptation
- High-dimensional sparse classification using exponential weighting with empirical hinge loss
- Adaptive posterior concentration rates for sparse high-dimensional linear regression with random design and unknown error variance
- On high-dimensional classification by sparse generalized Bayesian logistic regression
- Posterior and variational inference for deep neural networks with heavy-tailed weights
- Advances in Bayesian model selection consistency for high-dimensional generalized linear models
- Semiparametric posterior corrections
- On properties of fractional posterior in generalized reduced-rank regression
- On the convergence of coordinate ascent variational inference
- Concentration of a sparse Bayesian model with horseshoe prior in estimating high-dimensional precision matrix
- Misclassification excess risk bounds for 1-bit matrix completion
- Loss-Based Variational Bayes Prediction
- Concentration properties of fractional posterior in 1-bit matrix completion
- Unified algorithms for RL with decision-estimation coefficients: PAC, reward-free, preference-based learning and beyond
- Kullback-Leibler excess risk bounds for exponential weighted aggregation in generalized linear models
- Consistency of variational inference for Besov priors in non-linear inverse problems
- Variational Bayes neural networks for high-dimensional non parametric regression: Minimax optimality and adaptivity
- Bayesian inference in high-dimensional models
- Variational approximated restricted maximum likelihood estimation for spatial data
- Robust estimation with variational Bayes in presence of competing risks
This page was built for publication: Concentration of tempered posteriors and of their variational approximations
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2196229)
