Concentration of tempered posteriors and of their variational approximations

DOI10.1214/19-AOS1855zbMATH Open1454.62135arXiv1706.09293OpenAlexW3043271774MaRDI QIDQ2196229FDOQ2196229

Pierre Alquier, James Ridgway

Publication date: 28 August 2020

Published in: The Annals of Statistics (Search for Journal in Brave)

Abstract: While Bayesian methods are extremely popular in statistics and machine learning, their application to massive datasets is often challenging, when possible at all. Indeed, the classical MCMC algorithms are prohibitively slow when both the model dimension and the sample size are large. Variational Bayesian methods aim at approximating the posterior by a distribution in a tractable family. Thus, MCMC are replaced by an optimization algorithm which is orders of magnitude faster. VB methods have been applied in such computationally demanding applications as including collaborative filtering, image and video processing, NLP and text processing... However, despite very nice results in practice, the theoretical properties of these approximations are usually not known. In this paper, we propose a general approach to prove the concentration of variational approximations of fractional posteriors. We apply our theory to two examples: matrix completion, and Gaussian VB.

Full work available at URL: https://arxiv.org/abs/1706.09293

Recommendations

• Frequentist consistency of variational Bayes
• Convergence rates of variational posterior distributions
• \(\alpha\)-variational inference with statistical guarantees
• Consistency of variational Bayes inference for estimation and model selection in mixtures
• On the properties of variational approximations of Gibbs posteriors

zbMATH Keywords

rate of convergencevariational approximationPAC-Bayesian boundsconcentration of posterior

Mathematics Subject Classification ID

Parametric tolerance and confidence regions (62F25) Nonparametric tolerance and confidence regions (62G15) Bayesian problems; characterization of Bayes procedures (62C10) Minimax procedures in statistical decision theory (62C20)

Cites Work

• Title not available (Why is that?)
• Pattern recognition and machine learning.
• Bayesian fractional posteriors
• Title not available (Why is that?)
• Fundamentals of Nonparametric Bayesian Inference
• Introduction to nonparametric estimation
• Convergence rates of posterior distributions.
• Noisy low-rank matrix completion with general sampling distribution
• Nuclear-norm penalization and optimal rates for noisy low-rank matrix completion
• 1-bit matrix completion: PAC-Bayesian analysis of a variational approximation
• A Bayesian approach for noisy matrix completion: optimal rate under general sampling distribution
• The Power of Convex Relaxation: Near-Optimal Matrix Completion
• Inconsistency of Bayesian inference for misspecified linear models, and a proposal for repairing it
• Online Learning and Online Convex Optimization
• Statistical learning theory and stochastic optimization. Ecole d'Eté de Probabilitiés de Saint-Flour XXXI -- 2001.
• Rényi Divergence and Kullback-Leibler Divergence
• The Variational Gaussian Approximation Revisited
• On the properties of variational approximations of Gibbs posteriors
• Leave Pima Indians alone: binary regression as a benchmark for Bayesian computation
• The Safe Bayesian
• Estimation bounds and sharp oracle inequalities of regularized procedures with Lipschitz loss functions
• Frequentist Consistency of Variational Bayes
• Active Learning and Basis Selection for Kernel-Based Linear Models: A Bayesian Perspective

Cited In (21)

• Convergence rates of variational posterior distributions
• Variational Bayes for High-Dimensional Linear Regression With Sparse Priors
• Title not available (Why is that?)
• Tight risk bound for high dimensional time series completion
• Uncertainty quantification for sparse spectral variational approximations in Gaussian process regression
• On the properties of variational approximations of Gibbs posteriors
• Comprehensive study of variational Bayes classification for dense deep neural networks
• Statistical foundation of variational Bayes neural networks
• Heavy-tailed Bayesian nonparametric adaptation
• Online estimation and community detection of network point processes for event streams
• Adaptive variational Bayes: optimality, computation and applications
• On the approximation accuracy of Gaussian variational inference
• Local convexity of the TAP free energy and AMP convergence for \(\mathbb{Z}_2\)-synchronization
• The computational asymptotics of Gaussian variational inference and the Laplace approximation
• Frequentist Consistency of Variational Bayes
• Numerical comparisons between Bayesian and frequentist low-rank matrix completion: estimation accuracy and uncertainty quantification
• High-dimensional sparse classification using exponential weighting with empirical hinge loss
• Approximating Bayes in the 21st century
• Emerging directions in Bayesian computation
• Asymptotic consistency of loss-calibrated variational Bayes
• User-friendly Introduction to PAC-Bayes Bounds

Uses Software

• PRMLT

This page was built for publication: Concentration of tempered posteriors and of their variational approximations

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2196229)