The computational asymptotics of Gaussian variational inference and the Laplace approximation
From MaRDI portal
(Redirected from Publication:2172111)
Abstract: Gaussian variational inference and the Laplace approximation are popular alternatives to Markov chain Monte Carlo that formulate Bayesian posterior inference as an optimization problem, enabling the use of simple and scalable stochastic optimization algorithms. However, a key limitation of both methods is that the solution to the optimization problem is typically not tractable to compute; even in simple settings the problem is nonconvex. Thus, recently developed statistical guarantees -- which all involve the (data) asymptotic properties of the global optimum -- are not reliably obtained in practice. In this work, we provide two major contributions: a theoretical analysis of the asymptotic convexity properties of variational inference with a Gaussian family and the maximum a posteriori (MAP) problem required by the Laplace approximation; and two algorithms -- consistent Laplace approximation (CLA) and consistent stochastic variational inference (CSVI) -- that exploit these properties to find the optimal approximation in the asymptotic regime. Both CLA and CSVI involve a tractable initialization procedure that finds the local basin of the optimum, and CSVI further includes a scaled gradient descent algorithm that provably stays locally confined to that basin. Experiments on nonconvex synthetic and real-data examples show that compared with standard variational and Laplace approximations, both CSVI and CLA improve the likelihood of obtaining the global optimum of their respective optimization problems.
Recommendations
- Asymptotic normality and valid inference for Gaussian variational approximation
- Variational Bayesian approximation. A rigorous approach
- The Variational Gaussian Approximation Revisited
- On the properties of variational approximations of Gibbs posteriors
- Variational Bayesian inference with Gaussian-mixture approximations
- Stochastic complexities of Gaussian mixtures in variational Bayesian approximation
- Gauging variational inference
- A Measure-Theoretic Variational Bayesian Algorithm for Large Dimensional Problems
- Perturbative corrections for approximate inference in Gaussian latent variable models
Cites work
- scientific article; zbMATH DE number 6377992 (Why is no real title available?)
- scientific article; zbMATH DE number 3850830 (Why is no real title available?)
- scientific article; zbMATH DE number 3902413 (Why is no real title available?)
- scientific article; zbMATH DE number 4072103 (Why is no real title available?)
- scientific article; zbMATH DE number 44710 (Why is no real title available?)
- scientific article; zbMATH DE number 1222283 (Why is no real title available?)
- scientific article; zbMATH DE number 1324223 (Why is no real title available?)
- scientific article; zbMATH DE number 3438144 (Why is no real title available?)
- scientific article; zbMATH DE number 2107836 (Why is no real title available?)
- scientific article; zbMATH DE number 2117879 (Why is no real title available?)
- scientific article; zbMATH DE number 795286 (Why is no real title available?)
- scientific article; zbMATH DE number 7306861 (Why is no real title available?)
- scientific article; zbMATH DE number 7415111 (Why is no real title available?)
- scientific article; zbMATH DE number 3252891 (Why is no real title available?)
- A Stochastic Approximation Method
- A Study on Invariance of $f$-Divergence and Its Application to Speech Recognition
- A primer on monotone operator methods
- Adaptive subgradient methods for online learning and stochastic optimization
- Advanced Lectures on Machine Learning
- Asymptotic Properties of Non-Linear Least Squares Estimators
- Asymptotic Statistics
- Asymptotic equivalence of empirical likelihood and Bayesian MAP
- Asymptotic normality and valid inference for Gaussian variational approximation
- Automatic differentiation variational inference
- Concentration inequalities. A nonasymptotic theory of independence
- Concentration of tempered posteriors and of their variational approximations
- Convergence rates of posterior distributions.
- Convergence rates of variational posterior distributions
- Convex analysis and monotone operator theory in Hilbert spaces
- Convex optimization: algorithms and complexity
- Criteria for quasi-convexity and pseudo-convexity: Relationships and comparisons
- Frequentist consistency of variational Bayes
- General state space Markov chains and MCMC algorithms
- Gradient Convergence in Gradient methods with Errors
- Graphical models, exponential families, and variational inference
- Laplace approximation in high-dimensional Bayesian regression
- Local optima smoothing for global optimization
- Machine learning. A probabilistic perspective
- Markov chains and stochastic stability
- Maximum a posteriori estimators as a limit of Bayes estimators
- Minimization of functions having Lipschitz continuous first partial derivatives
- Monte Carlo sampling methods using Markov chains and their applications
- Numerical analysis. A mathematical introduction. Transl. from the French by John Taylor
- On Choosing and Bounding Probability Metrics
- On the convergence of the Laplace approximation and noise-level-robustness of Laplace-based Monte Carlo methods for Bayesian inverse problems
- Pattern recognition and machine learning.
- Quasi-Concave Programming
- Rates of convergence of posterior distributions.
- Rényi Divergence and Kullback-Leibler Divergence
- Sampling-Based Approaches to Calculating Marginal Densities
- Statistical guarantees for the EM algorithm: from population to sample-based analysis
- The Bernstein-von Mises theorem under misspecification
- Weak convergence and empirical processes. With applications to statistics
- What is invexity?
- \(\alpha\)-variational inference with statistical guarantees
Cited in
(6)- Optimizing Variational Representations of Divergences and Accelerating Their Statistical Estimation
- Amortized Variational Inference: A Systematic Review
- A Measure-Theoretic Variational Bayesian Algorithm for Large Dimensional Problems
- Asymptotic normality and valid inference for Gaussian variational approximation
- The Variational Gaussian Approximation Revisited
- On the properties of variational approximations of Gibbs posteriors
This page was built for publication: The computational asymptotics of Gaussian variational inference and the Laplace approximation
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2172111)
