Correlated variables in regression: clustering and sparse estimation

DOI10.1016/J.JSPI.2013.05.019zbMATH Open1278.62103arXiv1209.5908OpenAlexW2040912306MaRDI QIDQ394080FDOQ394080

Peter Bühlmann, Philipp Rütimann, Sara Van De Geer, Cun-Hui Zhang

Publication date: 24 January 2014

Published in: Journal of Statistical Planning and Inference (Search for Journal in Brave)

Abstract: We consider estimation in a high-dimensional linear model with strongly correlated variables. We propose to cluster the variables first and do subsequent sparse estimation such as the Lasso for cluster-representatives or the group Lasso based on the structure from the clusters. Regarding the first step, we present a novel and bottom-up agglomerative clustering algorithm based on canonical correlations, and we show that it finds an optimal solution and is statistically consistent. We also present some theoretical arguments that canonical correlation based clustering leads to a better-posed compatibility constant for the design matrix which ensures identifiability and an oracle inequality for the group Lasso. Furthermore, we discuss circumstances where cluster-representatives and using the Lasso as subsequent estimator leads to improved results for prediction and detection of variables. We complement the theoretical analysis with various empirical results.

Full work available at URL: https://arxiv.org/abs/1209.5908

zbMATH Keywords

Lassovariable selectionhigh-dimensional inferencegroup Lassohierarchical clusteringoracle inequalityvariable screeningcanonical correlations

Mathematics Subject Classification ID

Computational methods for problems pertaining to statistics (62-08) Classification and discrimination; cluster analysis (statistical aspects) (62H30) Linear regression; mixed models (62J05) Measures of association (correlation, canonical correlation, etc.) (62H20) Ridge regression; shrinkage estimators (Lasso) (62J07)

Cites Work

• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Simultaneous Regression Shrinkage, Variable Selection, and Supervised Clustering of Predictors with OSCAR
• High-dimensional additive modeling
• The Adaptive Lasso and Its Oracle Properties
• Statistics for high-dimensional data. Methods, theory and applications.
• Lasso-type recovery of sparse representations for high-dimensional data
• On the conditions used to prove oracle results for the Lasso
• Simultaneous analysis of Lasso and Dantzig selector
• High-dimensional graphs and variable selection with the Lasso
• Regularization and Variable Selection Via the Elastic Net
• Model Selection and Estimation in Regression with Grouped Variables
• The sparse Laplacian shrinkage estimator for high-dimensional regression
• Scaled sparse linear regression
• The sparsity and bias of the LASSO selection in high-dimensional linear regression
• Relaxed Lasso
• Local operator theory, random matrices and Banach spaces.
• Rejoinder: One-step sparse estimates in nonconcave penalized likelihood models
• Sparse regression with exact clustering
• PARTIAL CORRELATION AND CONDITIONAL CORRELATION AS MEASURES OF CONDITIONAL INDEPENDENCE
• The Lasso, correlated design, and improved oracle inequalities
• Finding predictive gene groups from microarray data

Cited In (27)

• On the Use of Minimum Penalties in Statistical Learning
• Numerical characterization of support recovery in sparse regression with correlated design
• Discussion of ``Correlated variables in regression: clustering and sparse estimation
• Bayesian latent factor on image regression with nonignorable missing data
• Hierarchical inference for genome-wide association studies: a view on methodology with software
• Graph-Based Regularization for Regression Problems with Alignment and Highly Correlated Designs
• Sequential knockoffs for continuous and categorical predictors: with application to a large psoriatic arthritis clinical trial pool
• Title not available (Why is that?)
• Weak signals in high‐dimensional regression: Detection, estimation and prediction
• The cluster graphical Lasso for improved estimation of Gaussian graphical models
• Evolution of high-frequency systematic trading: a performance-driven gradient boosting model
• Bayesian linear regression with sparse priors
• A clustering-based feature selection method for automatically generated relational attributes
• Split Regularized Regression
• A sequential rejection testing method for high-dimensional regression with correlated variables
• Group linear algorithm with sparse principal decomposition: a variable selection and clustering method for generalized linear models
• A General Framework for Estimation and Inference From Clusters of Features
• A Cluster Elastic Net for Multivariate Regression
• Modeling association between multivariate correlated outcomes and high-dimensional sparse covariates: the adaptive SVS method
• A component lasso
• Correlation and variable importance in random forests
• Robust grouped variable selection using distributionally robust optimization
• Extensions of stability selection using subsamples of observations and covariates
• Semi-Standard Partial Covariance Variable Selection When Irrepresentable Conditions Fail
• The Trimmed Lasso: Sparse Recovery Guarantees and Practical Optimization by the Generalized Soft-Min Penalty
• MCEN: a method of simultaneous variable selection and clustering for high-dimensional multinomial regression
• Spatially relaxed inference on high-dimensional linear models

Uses Software

• glasso
• glmnet
• hgam

This page was built for publication: Correlated variables in regression: clustering and sparse estimation

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q394080)