The computational asymptotics of Gaussian variational inference and the Laplace approximation

DOI10.1007/S11222-022-10125-YzbMATH Open1495.62015arXiv2104.05886OpenAlexW4292074665WikidataQ114223422 ScholiaQ114223422MaRDI QIDQ2172111FDOQ2172111

Authors: Zuheng Xu, Trevor Campbell 

Publication date: 15 September 2022

Published in: Statistics and Computing (Search for Journal in Brave)

Abstract: Gaussian variational inference and the Laplace approximation are popular alternatives to Markov chain Monte Carlo that formulate Bayesian posterior inference as an optimization problem, enabling the use of simple and scalable stochastic optimization algorithms. However, a key limitation of both methods is that the solution to the optimization problem is typically not tractable to compute; even in simple settings the problem is nonconvex. Thus, recently developed statistical guarantees -- which all involve the (data) asymptotic properties of the global optimum -- are not reliably obtained in practice. In this work, we provide two major contributions: a theoretical analysis of the asymptotic convexity properties of variational inference with a Gaussian family and the maximum a posteriori (MAP) problem required by the Laplace approximation; and two algorithms -- consistent Laplace approximation (CLA) and consistent stochastic variational inference (CSVI) -- that exploit these properties to find the optimal approximation in the asymptotic regime. Both CLA and CSVI involve a tractable initialization procedure that finds the local basin of the optimum, and CSVI further includes a scaled gradient descent algorithm that provably stays locally confined to that basin. Experiments on nonconvex synthetic and real-data examples show that compared with standard variational and Laplace approximations, both CSVI and CLA improve the likelihood of obtaining the global optimum of their respective optimization problems.

Full work available at URL: https://arxiv.org/abs/2104.05886

Recommendations

• Asymptotic normality and valid inference for Gaussian variational approximation
• Variational Bayesian approximation. A rigorous approach
• The Variational Gaussian Approximation Revisited
• On the properties of variational approximations of Gibbs posteriors
• Variational Bayesian inference with Gaussian-mixture approximations
• Stochastic complexities of Gaussian mixtures in variational Bayesian approximation
• Gauging variational inference
• A Measure-Theoretic Variational Bayesian Algorithm for Large Dimensional Problems
• Perturbative corrections for approximate inference in Gaussian latent variable models

zbMATH Keywords

Bayesian statisticsLaplace approximationvariational inferenceBernstein-von Misescomputational asymptotics

Mathematics Subject Classification ID

Computational methods for problems pertaining to statistics (62-08) Bayesian inference (62F15) Convex programming (90C25)

Cites Work

• Automatic differentiation variational inference
• Adaptive subgradient methods for online learning and stochastic optimization
• Weak convergence and empirical processes. With applications to statistics
• Graphical models, exponential families, and variational inference
• Title not available (Why is that?)
• Title not available (Why is that?)
• Pattern recognition and machine learning.
• Convex analysis and monotone operator theory in Hilbert spaces
• Machine learning. A probabilistic perspective
• Title not available (Why is that?)
• Title not available (Why is that?)
• Sampling-Based Approaches to Calculating Marginal Densities
• Title not available (Why is that?)
• Asymptotic Statistics
• On Choosing and Bounding Probability Metrics
• Title not available (Why is that?)
• Concentration inequalities. A nonasymptotic theory of independence
• Asymptotic Properties of Non-Linear Least Squares Estimators
• Monte Carlo sampling methods using Markov chains and their applications
• A Stochastic Approximation Method
• Convergence rates of posterior distributions.
• Concentration of tempered posteriors and of their variational approximations
• Title not available (Why is that?)
• Markov chains and stochastic stability
• General state space Markov chains and MCMC algorithms
• Title not available (Why is that?)
• Statistical guarantees for the EM algorithm: from population to sample-based analysis
• Rates of convergence of posterior distributions.
• Minimization of functions having Lipschitz continuous first partial derivatives
• Quasi-Concave Programming
• What is invexity?
• Title not available (Why is that?)
• Title not available (Why is that?)
• Gradient Convergence in Gradient methods with Errors
• The Bernstein-von Mises theorem under misspecification
• Title not available (Why is that?)
• Local optima smoothing for global optimization
• Rényi Divergence and Kullback-Leibler Divergence
• Convex optimization: algorithms and complexity
• Title not available (Why is that?)
• Numerical analysis. A mathematical introduction. Transl. from the French by John Taylor
• Title not available (Why is that?)
• Advanced Lectures on Machine Learning
• Criteria for quasi-convexity and pseudo-convexity: Relationships and comparisons
• A Study on Invariance of $f$-Divergence and Its Application to Speech Recognition
• Asymptotic normality and valid inference for Gaussian variational approximation
• Asymptotic equivalence of empirical likelihood and Bayesian MAP
• A primer on monotone operator methods
• Laplace approximation in high-dimensional Bayesian regression
• Maximum a posteriori estimators as a limit of Bayes estimators
• Convergence rates of variational posterior distributions
• On the convergence of the Laplace approximation and noise-level-robustness of Laplace-based Monte Carlo methods for Bayesian inverse problems
• \(\alpha\)-variational inference with statistical guarantees
• Frequentist consistency of variational Bayes
• Title not available (Why is that?)

Cited In (6)

• The Variational Gaussian Approximation Revisited
• On the properties of variational approximations of Gibbs posteriors
• Amortized Variational Inference: A Systematic Review
• Optimizing Variational Representations of Divergences and Accelerating Their Statistical Estimation
• A Measure-Theoretic Variational Bayesian Algorithm for Large Dimensional Problems
• Asymptotic normality and valid inference for Gaussian variational approximation

Uses Software

• PMTK
• AdaGrad
• Adam
• ADVI

This page was built for publication: The computational asymptotics of Gaussian variational inference and the Laplace approximation

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2172111)