Projective Integral Updates for High-Dimensional Variational Inference

DOI10.1137/22M1529919arXiv2301.08374OpenAlexW4391645332WikidataQ128381766 ScholiaQ128381766MaRDI QIDQ6131419

Jed A. Duersch

Publication date: 5 April 2024

Published in: SIAM/ASA Journal on Uncertainty Quantification (Search for Journal in Brave)

Abstract: This work proposes a quasirandom sequence of quadratures for high-dimensional mean-field variational inference and a related sparsifying methodology. Each iterate of the sequence contains two evaluations points that combine to correctly integrate all univariate quadratic functions, as well as univariate cubics if the mean-field factors are symmetric. More importantly, averaging results over short subsequences achieves periodic exactness on a much larger space of multivariate polynomials of quadratic total degree. This framework is devised by first considering stochastic blocked mean-field quadratures, which may be useful in other contexts. By replacing pseudorandom sequences with quasirandom sequences, over half of all multivariate quadratic basis functions integrate exactly with only 4 function evaluations, and the exactness dimension increases for longer subsequences. Analysis shows how these efficient integrals characterize the dominant log-posterior contributions to mean-field variational approximations, including diagonal Hessian approximations, to support a robust sparsifying methodology in deep learning algorithms. A numerical demonstration of this approach on a simple Convolutional Neural Network for MNIST retains high test accuracy, 96.9%, while training over 98.9% of parameters to zero in only 10 epochs, bearing potential to reduce both storage and energy requirements for deep learning models.

Full work available at URL: https://arxiv.org/abs/2301.08374

zbMATH Keywords

cubaturequadraturequasi-Newtonvariational inferencespike-and-slabHessian approximationHadamard basisGaussian mean-field

Mathematics Subject Classification ID

Computational methods for problems pertaining to statistics (62-08) Parametric inference under constraints (62F30) Bayesian inference (62F15) Monte Carlo methods (65C05) Numerical optimization and variational techniques (65K10) Numerical quadrature and cubature formulas (65D32)

Cites Work

• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Numerical integration using sparse grids
• Smolyak cubature of given polynomial degree with few nodes for increasing dimension
• Simple cubature formulas with high polynomial exactness
• Generalized parallel tempering on Bayesian inverse problems
• An estimator for the diagonal of a matrix
• Construction of fully symmetric numerical integration formulas
• A Systematization of the Unscented Kalman Filter Theory
• Exactness of Quadrature Formulas
• BAYESIAN INFERENCE OF STOCHASTIC REACTION NETWORKS USING MULTIFIDELITY SEQUENTIAL TEMPERED MARKOV CHAIN MONTE CARLO
• Cubature, Approximation, and Isotropy in the Hypercube
• High-dimensional integration: The quasi-Monte Carlo way
• Algorithms for Numerical Analysis in High Dimensions
• On Information and Sufficiency
• A Stochastic Approximation Method

This page was built for publication: Projective Integral Updates for High-Dimensional Variational Inference