VAE-KRnet and its applications to variational Bayes

DOI10.4208/CICP.OA-2021-0087zbMATH Open1486.62096arXiv2006.16431OpenAlexW4220785892MaRDI QIDQ5077693FDOQ5077693

Authors: Xiaoliang Wan, Shuangqing Wei 

Publication date: 19 May 2022

Published in: Communications in Computational Physics (Search for Journal in Brave)

Abstract: In this work, we have proposed a generative model, called VAE-KRnet, for density estimation or approximation, which combines the canonical variational autoencoder (VAE) with our recently developed flow-based generative model, called KRnet. VAE is used as a dimension reduction technique to capture the latent space, and KRnet is used to model the distribution of the latent variable. Using a linear model between the data and the latent variable, we show that VAE-KRnet can be more effective and robust than the canonical VAE. VAE-KRnet can be used as a density model to approximate either data distribution or an arbitrary probability density function (PDF) known up to a constant. VAE-KRnet is flexible in terms of dimensionality. When the number of dimensions is relatively small, KRnet can effectively approximate the distribution in terms of the original random variable. For high-dimensional cases, we may use VAE-KRnet to incorporate dimension reduction. One important application of VAE-KRnet is the variational Bayes for the approximation of the posterior distribution. The variational Bayes approaches are usually based on the minimization of the Kullback-Leibler (KL) divergence between the model and the posterior. For high-dimensional distributions, it is very challenging to construct an accurate density model due to the curse of dimensionality, where extra assumptions are often introduced for efficiency. For instance, the classical mean-field approach assumes mutual independence between dimensions, which often yields an underestimated variance due to oversimplification. To alleviate this issue, we include into the loss the maximization of the mutual information between the latent random variable and the original random variable, which helps keep more information from the region of low density such that the estimation of variance is improved.

Full work available at URL: https://arxiv.org/abs/2006.16431

Recommendations

• Concentration of tempered posteriors and of their variational approximations
• scientific article; zbMATH DE number 6982332
• Convergence rates of variational posterior distributions
• Deep variational inference
• On variational Bayes estimation and variational information criteria for linear regression models

zbMATH Keywords

deep learningvariational Bayesuncertainty quantificationBayesian inverse problemsgenerative modeling

Mathematics Subject Classification ID

Computational methods for problems pertaining to statistics (62-08) Density estimation (62G07) Artificial neural networks and deep learning (68T07) Bayesian problems; characterization of Bayes procedures (62C10)

Cites Work

• Multivariate Density Estimation
• Title not available (Why is that?)
• From Knothe's Transport to Brenier's Map and a Continuation Method for Optimal Transport
• Hybrid Monte Carlo on Hilbert spaces
• Multilevel Monte Carlo Methods
• Adaptive deep density approximation for Fokker-Planck equations
• Coupling the reduced-order model and the generative model for an importance sampling estimator

Cited In (4)

• Physics-informed neural networks for approximating dynamic (hyperbolic) PDEs of second order in time: error analysis and algorithms
• On computing the hyperparameter of extreme learning machines: algorithm and application to computational PDEs, and comparison with classical and high-order finite elements
• DAS-PINNs: a deep adaptive sampling method for solving high-dimensional partial differential equations
• Numerical computation of partial differential equations by hidden-layer concatenated extreme learning machine

Uses Software

• Adam
• NICE
• Glow
• torchdiffeq
• ANODEs
• Wasserstein GAN
• Flow-GAN

This page was built for publication: VAE-KRnet and its applications to variational Bayes

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5077693)