The variational Garrote

DOI10.1007/S10994-013-5427-7zbMATH Open1317.68167DBLPjournals/ml/KappenG14OpenAlexW2124558419WikidataQ59321114 ScholiaQ59321114MaRDI QIDQ479478FDOQ479478

Authors Hilbert J. Kappen, Vicenç Gómez

Publication date 5 December 2014

Published in Machine Learning (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1109.0486

zbMATH Keywords

spike-and-slabsparse regressionmean-fieldvariational approximationnonnegative garrote

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05)

Abstract: In this paper, we present a new variational method for sparse regression using $L_0$ regularization. The variational parameters appear in the approximate model in a way that is similar to Breiman's Garrote model. We refer to this method as the variational Garrote (VG). We show that the combination of the variational approximation and $L_0$ regularization has the effect of making the problem effectively of maximal rank even when the number of samples is small compared to the number of variables. The VG is compared numerically with the Lasso method, ridge regression and the recently introduced paired mean field method (PMF) (M. Titsias & M. L'azaro-Gredilla., NIPS 2012). Numerical results show that the VG and PMF yield more accurate predictions and more accurately reconstruct the true model than the other methods. It is shown that the VG finds correct solutions when the Lasso solution is inconsistent due to large input correlations. Globally, VG is significantly faster than PMF and tends to perform better as the problems become denser and in problems with strongly correlated inputs. The naive implementation of the VG scales cubic with the number of features. By introducing Lagrange multipliers we obtain a dual formulation of the problem that scales cubic in the number of samples, but close to linear in the number of features.

Recommendations

• A variational Bayes approach to variable selection
• Sparse Online Variational Bayesian Regression
• Adaptive sparse regression
• Variational Bayes for High-Dimensional Linear Regression With Sparse Priors
• On the Non-Negative Garrotte Estimator

Cites work

• scientific article; zbMATH DE number 5957408 (Why is no real title available?)
• scientific article; zbMATH DE number 845714 (Why is no real title available?)
• A Statistical View of Some Chemometrics Regression Tools
• A review of Bayesian variable selection methods: what, how and which
• Advanced mean field methods. Theory and practice
• An introduction to variational methods for graphical models
• Bayesian Variable Selection in Linear Regression
• Bayesian hyper-Lassos with non-convex penalization
• Bayesian learning in sparse graphical factor models via variational mean-field annealing
• Better Subset Regression Using the Nonnegative Garrote
• Compressed sensing
• Decoding by Linear Programming
• Graphical models, exponential families, and variational inference
• Model uncertainty
• Multivariate Bayesian Variable Selection and Prediction
• Regression Shrinkage and Selection via The Lasso: A Retrospective
• Scalable variational inference for Bayesian variable selection in regression, and its accuracy in genetic association studies
• SparseNet: coordinate descent with nonconvex penalties
• Spike and slab variable selection: frequentist and Bayesian strategies
• The Bayesian Lasso
• The periodic solution of two-dimensional linear nonstationary systems and estimation of the attraction domain boundary in the problem of control of a wheeled robot
• Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties

Cited in

(2)

• Estimating large‐dimensional connectedness tables: The great moderation through the lens of sectoral spillovers
• Sparsity constrained estimation in image processing and computer vision

Describes a project that uses

Uses Software

• glmnet
• sparsenet

This page was built for publication: The variational Garrote

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q479478)