Infinite-dimensional gradient-based descent for alpha-divergence minimisation

DOI10.1214/20-AOS2035MaRDI QIDQ2054493zbMATH OpenOpenAlexFDO

Authors Kamélia Daudel, Randal Douc, François Portier

Publication date 3 December 2021

Published in The Annals of Statistics (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2005.10618, https://projecteuclid.org/journals/annals-of-statistics/volume-49/issue-4/Infinite-dimensional-gradient-based-descent-for-alpha-divergence-minimisation/10.1214/20-AOS2035.full

zbMATH Keywords

variational inferenceKullback-Leibler divergencemirror descentalpha-divergence

Mathematics Subject Classification ID

Bayesian inference (62F15) Density estimation (62G07) Parametric inference under constraints (62F30) Robustness and adaptive procedures (parametric inference) (62F35) Sequential statistical methods (62L99)

Abstract: This paper introduces the $(alpha,Gamma)$-descent, an iterative algorithm which operates on measures and performs $alpha$-divergence minimisation in a Bayesian framework. This gradient-based procedure extends the commonly-used variational approximation by adding a prior on the variational parameters in the form of a measure. We prove that for a rich family of functions $Gamma$, this algorithm leads at each step to a systematic decrease in the $alpha$-divergence and derive convergence results. Our framework recovers the Entropic Mirror Descent algorithm and provides an alternative algorithm that we call the Power Descent. Moreover, in its stochastic formulation, the $(alpha,Gamma)$-descent allows to optimise the mixture weights of any given mixture model without any information on the underlying distribution of the variational parameters. This renders our method compatible with many choices of parameters updates and applicable to a wide range of Machine Learning tasks. We demonstrate empirically on both toy and real-world examples the benefit of using the Power descent and going beyond the Entropic Mirror Descent framework, which fails as the dimension grows.

Recommendations

• Integration of Stochastic Models by Minimizing α-Divergence
• The information geometry of mirror descent
• Learning continuous latent variable models with Bregman divergences
• A gradient descent perspective on Sinkhorn
• Unifying Divergence Minimization and Statistical Inference Via Convex Duality

Cites work

• scientific article; zbMATH DE number 6377992 (Why is no real title available?)
• scientific article; zbMATH DE number 3173999 (Why is no real title available?)
• scientific article; zbMATH DE number 1222284 (Why is no real title available?)
• scientific article; zbMATH DE number 3200971 (Why is no real title available?)
• 10.1162/jmlr.2003.3.4-5.993
• A Stochastic Approximation Method
• Adaptive importance sampling in monte carlo integration
• An introduction to variational methods for graphical models
• Bayesian Estimates of Equation System Parameters: An Application of Integration by Monte Carlo
• Central limit theorem for sequential Monte Carlo methods and its application to Bayesian inference
• Convergence of adaptive mixtures of importance sampling schemes
• Convex optimization: algorithms and complexity
• Efficiency versus robustness: The case for minimum Hellinger distance and related methods
• Expectation Propagation in the Large Data Limit
• Families of alpha-, beta- and gamma-divergences: flexible and robust measures of similarities
• Large-scale machine learning with stochastic gradient descent
• Markov Processes and the H-Theorem
• Mirror descent and nonlinear projected subgradient methods for convex optimization.
• On Information and Sufficiency
• Optimal global rates of convergence for nonparametric regression
• Prox-Method with Rate of Convergence O(1/t) for Variational Inequalities with Lipschitz Continuous Monotone Operators and Smooth Convex-Concave Saddle Point Problems
• Robust Stochastic Approximation Approach to Stochastic Programming
• Rényi Divergence and Kullback-Leibler Divergence
• Safe adaptive importance sampling: a mixture approach

Cited in

(3)

• On the approximation accuracy of Gaussian variational inference
• The recursive variational Gaussian approximation (R-VGA)
• Alpha-divergence minimization for deep Gaussian processes

This page was built for publication: Infinite-dimensional gradient-based descent for alpha-divergence minimisation

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2054493)