Sparse Online Variational Bayesian Regression

DOI10.1137/21M1401188MaRDI QIDQ5052896zbMATH OpenOpenAlexFDO

Authors Vitaly Zankin, K. J. H. Law

Publication date 25 November 2022

Published in SIAM/ASA Journal on Uncertainty Quantification (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2102.12261

zbMATH Keywords

expectation maximizationsparsity-promoting priorvariational Bayesian regression

Mathematics Subject Classification ID

Bayesian problems; characterization of Bayes procedures (62C10) Linear regression; mixed models (62J05) Ridge regression; shrinkage estimators (Lasso) (62J07) Numerical optimization and variational techniques (65K10)

Abstract: This work considers variational Bayesian inference as an inexpensive and scalable alternative to a fully Bayesian approach in the context of sparsity-promoting priors. In particular, the priors considered arise from scale mixtures of Normal distributions with a generalized inverse Gaussian mixing distribution. This includes the variational Bayesian LASSO as an inexpensive and scalable alternative to the Bayesian LASSO introduced in [65]. It also includes a family of priors which more strongly promote sparsity. For linear models the method requires only the iterative solution of deterministic least squares problems. Furthermore, for p unknown covariates the method can be implemented exactly online with a cost of $O(p^3)$ in computation and $O(p^2)$ in memory per iteration -- in other words, the cost per iteration is independent of n, and in principle infinite data can be considered. For large $p$ an approximation is able to achieve promising results for a cost of $O(p)$ per iteration, in both computation and memory. Strategies for hyper-parameter tuning are also considered. The method is implemented for real and simulated data. It is shown that the performance in terms of variable selection and uncertainty quantification of the variational Bayesian LASSO can be comparable to the Bayesian LASSO for problems which are tractable with that method, and for a fraction of the cost. The present method comfortably handles $n=65536$, $p=131073$ on a laptop in less than 30 minutes, and $n=10^5$, $p=2.1imes10^6$ overnight.

Recommendations

• Online Sparse Gaussian Process Regression and Its Applications
• Sparse on-line Gaussian processes
• Variational Bayes for High-Dimensional Linear Regression With Sparse Priors
• Online Hyperparameter-Free Sparse Estimation Method
• A Variational Bayes Framework for Sparse Adaptive Estimation
• Variable Screening for Sparse Online Regression
• Online model selection based on the variational Bayes
• On the sparse Bayesian learning of linear models
• Online sparse identification for regression models

Cites work

• 10.1162/15324430152748236
• A dual algorithm for the solution of nonlinear variational problems via finite element approximation
• A randomized Kaczmarz algorithm with exponential convergence
• An Algorithm for the Machine Calculation of Complex Fourier Series
• An iterative thresholding algorithm for linear inverse problems with a sparsity constraint
• Atomic decomposition by basis pursuit
• Automatic differentiation variational inference
• Bayesian computation: a summary of the current state, and samples backwards and forwards
• Bayesian hyper-Lassos with non-convex penalization
• Bayesian inverse problems with l₁ priors: a randomize-then-optimize approach
• Bayesian learning for neural networks
• Bayesian linear regression with sparse priors
• Bayesian Variable Selection in Linear Regression
• Can one use total variation prior for edge-preserving Bayesian inversion?
• Cauchy difference priors for edge-preserving Bayesian inversion
• Compressed sensing
• Constructing least-squares polynomial approximations
• Correction
• Data Assimilation
• Discrete least squares polynomial approximation with random evaluations - application to parametric and stochastic elliptic PDEs
• Distributed optimization and statistical learning via the alternating direction method of multipliers
• Edge-preserving and scale-dependent properties of total variation regularization
• Fast and Exact Simulation of Stationary Gaussian Processes through Circulant Embedding of the Covariance Matrix
• Fast Markov chain Monte Carlo sampling for sparse Bayesian inference in high-dimensional inverse problems using L1-type priors
• Handbook of uncertainty quantification. In 2 volumes
• Hierachical Bayesian models and sparsity: \(\ell_2\)-magic
• scientific article; zbMATH DE number 3558383 (Why is no real title available?)
• scientific article; zbMATH DE number 3567782 (Why is no real title available?)
• scientific article; zbMATH DE number 1972910 (Why is no real title available?)
• scientific article; zbMATH DE number 3442988 (Why is no real title available?)
• scientific article; zbMATH DE number 2106098 (Why is no real title available?)
• scientific article; zbMATH DE number 2117879 (Why is no real title available?)
• scientific article; zbMATH DE number 845714 (Why is no real title available?)
• scientific article; zbMATH DE number 5060482 (Why is no real title available?)
• scientific article; zbMATH DE number 3227378 (Why is no real title available?)
• scientific article; zbMATH DE number 3273551 (Why is no real title available?)
• Inverse problems: a Bayesian perspective
• Iterative Methods for Total Variation Denoising
• Iteratively reweighted least squares minimization for sparse recovery
• Krylov Subspace Methods for Solving Large Unsymmetric Linear Systems
• Least angle regression. (With discussion)
• Linear convergence of iterative soft-thresholding
• Maximum-a-posteriori estimation with Bayesian confidence regions
• Mean field variational Bayes for continuous sparse signal shrinkage: pitfalls and remedies
• Methods of conjugate gradients for solving linear systems
• Nearly optimal Bayesian shrinkage for high-dimensional regression
• Nonparametric Bayesian data analysis
• Pattern recognition and machine learning.
• Proximal Markov chain Monte Carlo algorithms
• Ridge Regression: Biased Estimation for Nonorthogonal Problems
• Robust regression using iteratively reweighted least-squares
• Selection of tuning parameters, solution paths and standard errors for Bayesian Lassos
• Sequential Monte Carlo Methods in Practice
• Sparsity-Cognizant Total Least-Squares for Perturbed Compressive Sampling
• Spike-and-Slab Meets LASSO: A Review of the Spike-and-Slab LASSO
• The Bayesian Bridge
• The Bayesian Lasso
• The horseshoe estimator for sparse signals
• The spike-and-slab LASSO
• Variable Selection in Finite Mixture of Regression Models
• Variance prior forms for high-dimensional Bayesian variable selection

Cited in

(13)

• Variational Bayesian least squares: an application to brain-machine interface data
• Updating variational Bayes: fast sequential posterior inference
• Online model selection based on the variational Bayes
• Online Hyperparameter-Free Sparse Estimation Method
• The variational Garrote
• A Variational Inference Approach to Inverse Problems with Gamma Hyperpriors
• Variational Bayes for High-Dimensional Linear Regression With Sparse Priors
• Variational inference for Bayesian bridge regression
• Hierarchical ensemble Kalman methods with sparsity-promoting generalized gamma hyperpriors
• Contrast source inversion of sparse targets through multi-resolution Bayesian compressive sensing
• Sparse Bayesian learning using TMB (Template Model Builder)
• Bayesian inversion with Student's t priors based on Gaussian scale mixtures
• Mixtures of Gaussian process experts with \(\mathrm{SMC}^2\)

Describes a project that uses

Uses Software

• PDCO
• ADVI

This page was built for publication: Sparse Online Variational Bayesian Regression

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5052896)