Optimizing Variational Representations of Divergences and Accelerating Their Statistical Estimation
From MaRDI portal
Abstract: Variational representations of divergences and distances between high-dimensional probability distributions offer significant theoretical insights and practical advantages in numerous research areas. Recently, they have gained popularity in machine learning as a tractable and scalable approach for training probabilistic models and for statistically differentiating between data distributions. Their advantages include: 1) They can be estimated from data as statistical averages. 2) Such representations can leverage the ability of neural networks to efficiently approximate optimal solutions in function spaces. However, a systematic and practical approach to improving the tightness of such variational formulas, and accordingly accelerate statistical learning and estimation from data, is currently lacking. Here we develop such a methodology for building new, tighter variational representations of divergences. Our approach relies on improved objective functionals constructed via an auxiliary optimization problem. Furthermore, the calculation of the functional Hessian of objective functionals unveils the local curvature differences around the common optimal variational solution; this quantifies and orders the tightness gains between different variational representations. Finally, numerical simulations utilizing neural network optimization demonstrate that tighter representations can result in significantly faster learning and more accurate estimation of divergences in both synthetic and real datasets (of more than 1000 dimensions), often accelerated by nearly an order of magnitude.
Recommendations
- Variational representations and neural network estimation of Rényi divergences
- Gauging variational inference
- Estimating Divergence Functionals and the Likelihood Ratio by Convex Risk Minimization
- Divergence measures and a general framework for local variational approximation
- scientific article; zbMATH DE number 6377992
- Variational inference in nonconjugate models
- Unifying Divergence Minimization and Statistical Inference Via Convex Duality
- \(\alpha\)-variational inference with statistical guarantees
- The computational asymptotics of Gaussian variational inference and the Laplace approximation
- Fast and accurate variational inference for models with many latent variables
Cited in
(4)- (f,)-divergences: interpolating between f-divergences and integral probability metrics
- Amortized Variational Inference: A Systematic Review
- Lipschitz-regularized gradient flows and generative particle algorithms for high-dimensional scarce data
- On regularization schemes for data-driven optimization based on Cressie–Read divergence and CVaR
This page was built for publication: Optimizing Variational Representations of Divergences and Accelerating Their Statistical Estimation
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5088576)
