Infinite-dimensional gradient-based descent for alpha-divergence minimisation

DOI10.1214/20-AOS2035zbMATH Open1486.62065arXiv2005.10618OpenAlexW3204854592MaRDI QIDQ2054493FDOQ2054493

Authors: Kamélia Daudel, Randal Douc, François Portier 

Publication date: 3 December 2021

Published in: The Annals of Statistics (Search for Journal in Brave)

Abstract: This paper introduces the $(alpha,Gamma)$-descent, an iterative algorithm which operates on measures and performs $alpha$-divergence minimisation in a Bayesian framework. This gradient-based procedure extends the commonly-used variational approximation by adding a prior on the variational parameters in the form of a measure. We prove that for a rich family of functions $Gamma$, this algorithm leads at each step to a systematic decrease in the $alpha$-divergence and derive convergence results. Our framework recovers the Entropic Mirror Descent algorithm and provides an alternative algorithm that we call the Power Descent. Moreover, in its stochastic formulation, the $(alpha,Gamma)$-descent allows to optimise the mixture weights of any given mixture model without any information on the underlying distribution of the variational parameters. This renders our method compatible with many choices of parameters updates and applicable to a wide range of Machine Learning tasks. We demonstrate empirically on both toy and real-world examples the benefit of using the Power descent and going beyond the Entropic Mirror Descent framework, which fails as the dimension grows.

Full work available at URL: https://arxiv.org/abs/2005.10618

Recommendations

• Integration of Stochastic Models by Minimizing α-Divergence
• The information geometry of mirror descent
• Learning continuous latent variable models with Bregman divergences
• A gradient descent perspective on Sinkhorn
• Unifying Divergence Minimization and Statistical Inference Via Convex Duality

zbMATH Keywords

variational inferenceKullback-Leibler divergencemirror descentalpha-divergence

Mathematics Subject Classification ID

Bayesian inference (62F15) Density estimation (62G07) Parametric inference under constraints (62F30) Robustness and adaptive procedures (parametric inference) (62F35) Sequential statistical methods (62L99)

Cites Work

• On Information and Sufficiency
• Efficiency versus robustness: The case for minimum Hellinger distance and related methods
• Central limit theorem for sequential Monte Carlo methods and its application to Bayesian inference
• Title not available (Why is that?)
• A Stochastic Approximation Method
• Optimal global rates of convergence for nonparametric regression
• Mirror descent and nonlinear projected subgradient methods for convex optimization.
• Robust Stochastic Approximation Approach to Stochastic Programming
• Prox-Method with Rate of Convergence O(1/t) for Variational Inequalities with Lipschitz Continuous Monotone Operators and Smooth Convex-Concave Saddle Point Problems
• 10.1162/jmlr.2003.3.4-5.993
• Title not available (Why is that?)
• Title not available (Why is that?)
• Bayesian Estimates of Equation System Parameters: An Application of Integration by Monte Carlo
• An introduction to variational methods for graphical models
• Families of alpha-, beta- and gamma-divergences: flexible and robust measures of similarities
• Rényi Divergence and Kullback-Leibler Divergence
• Markov Processes and the H-Theorem
• Convex optimization: algorithms and complexity
• Large-scale machine learning with stochastic gradient descent
• Convergence of adaptive mixtures of importance sampling schemes
• Adaptive importance sampling in monte carlo integration
• Title not available (Why is that?)
• Safe adaptive importance sampling: a mixture approach
• Expectation Propagation in the Large Data Limit

Cited In (3)

• On the approximation accuracy of Gaussian variational inference
• The recursive variational Gaussian approximation (R-VGA)
• Alpha-divergence minimization for deep Gaussian processes

This page was built for publication: Infinite-dimensional gradient-based descent for alpha-divergence minimisation

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2054493)