Support union recovery in high-dimensional multivariate regression

DOI10.1214/09-AOS776MaRDI QIDQ2429923zbMATH OpenFDO

Authors Guillaume Obozinski, Martin J. Wainwright, Michael Jordan

Publication date 5 April 2011

Published in The Annals of Statistics (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/0808.0711

zbMATH Keywords

block-normgroup Lassohigh-dimensional scalingLassomultivariate regressionsecond-order cone programsimultaneous Lassosparsityvariable selection

Mathematics Subject Classification ID

Statistical ranking and selection procedures (62F07) Ridge regression; shrinkage estimators (Lasso) (62J07)

Abstract: In multivariate regression, a $K$-dimensional response vector is regressed upon a common set of $p$ covariates, with a matrix $B^{\ast }inmathbb{R}^{pimesK}$ of regression coefficients. We study the behavior of the multivariate group Lasso, in which block regularization based on the $el{l}_1/el{l}_2$ norm is used for support union recovery, or recovery of the set of $s$ rows for which $B^{\ast }$ is nonzero. Under high-dimensional scaling, we show that the multivariate group Lasso exhibits a threshold for the recovery of the exact row pattern with high probability over the random design and noise that is specified by the sample complexity parameter $heta(n,p,s):=n/[2psi(B^{\ast })log(p-s)]$. Here $n$ is the sample size, and $psi(B^{\ast })$ is a sparsity-overlap function measuring a combination of the sparsities and overlaps of the $K$-regression coefficient vectors that constitute the model. We prove that the multivariate group Lasso succeeds for problem sequences $(n,p,s)$ such that $heta(n,p,s)$ exceeds a critical level $het{a}_u$, and fails for sequences such that $heta(n,p,s)$ lies below a critical level $het{a}_{ell}$. For the special case of the standard Gaussian ensemble, we show that $het{a}_{ell}=het{a}_u$ so that the characterization is sharp. The sparsity-overlap function $psi(B^{\ast })$ reveals that, if the design is uncorrelated on the active rows, $el{l}_1/el{l}_2$ regularization for multivariate regression never harms performance relative to an ordinary Lasso approach and can yield substantial improvements in sample complexity (up to a factor of $K$) when the coefficient vectors are suitably orthogonal. For more general designs, it is possible for the ordinary Lasso to outperform the multivariate group Lasso. We complement our analysis with simulations that demonstrate the sharpness of our theoretical results, even for relatively small problems.

Recommendations

• Multivariate sparse group Lasso for the multivariate multiple linear regression with an arbitrary group structure
• scientific article; zbMATH DE number 5056247
• Theoretical properties of the overlapping groups Lasso
• Oracle inequalities and optimal inference under group sparsity
• Consistency of the group Lasso and multiple kernel learning

Cites work

• A new approach to variable selection in least squares problems
• A Statistical View of Some Chemometrics Regression Tools
• Adaptive estimation of a quadratic functional by model selection.
• Asymptotic properties of bridge estimators in sparse high-dimensional regression models
• Asymptotics for Lasso-type estimators.
• Atomic Decomposition by Basis Pursuit
• Concentration inequalities and model selection. Ecole d'Eté de Probabilités de Saint-Flour XXXIII -- 2003.
• Consistency of the group Lasso and multiple kernel learning
• High-dimensional graphs and variable selection with the Lasso
• scientific article; zbMATH DE number 5957408 (Why is no real title available?)
• scientific article; zbMATH DE number 4062374 (Why is no real title available?)
• scientific article; zbMATH DE number 49190 (Why is no real title available?)
• scientific article; zbMATH DE number 1206370 (Why is no real title available?)
• scientific article; zbMATH DE number 2107836 (Why is no real title available?)
• scientific article; zbMATH DE number 845714 (Why is no real title available?)
• Information-Theoretic Limits on Sparsity Recovery in the High-Dimensional and Noisy Setting
• Just relax: convex programming methods for identifying sparse signals in noise
• Lasso-type recovery of sparse representations for high-dimensional data
• Least angle regression. (With discussion)
• Local operator theory, random matrices and Banach spaces.
• Model Selection and Estimation in Regression with Grouped Variables
• Sharp Thresholds for High-Dimensional and Noisy Sparsity Recovery Using $\ell _{1}$-Constrained Quadratic Programming (Lasso)
• Simultaneous analysis of Lasso and Dantzig selector
• The benefit of group sparsity
• The composite absolute penalties family for grouped and hierarchical variable selection
• The Group Lasso for Logistic Regression
• Uncertainty principles and ideal atomic decomposition
• Variable selection for multicategory SVM via adaptive sup-norm regularization
• Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties

Cited in

(only showing first 100 items - show all)

• Variable and boundary selection for functional data via multiclass logistic regression modeling
• Structured variable selection via prior-induced hierarchical penalty functions
• On the oracle property of a generalized adaptive elastic-net for multivariate linear regression with a diverging number of parameters
• Sparse-group independent component analysis with application to yield curves prediction
• Learning sparse FRAME models for natural image patterns
• Inference under Fine-Gray competing risks model with high-dimensional covariates
• Double fused Lasso penalized LAD for matrix regression
• High-order evaluation complexity for convexly-constrained optimization with non-Lipschitzian group sparsity terms
• High-dimensional index volatility models via Stein's identity
• Double fused Lasso regularized regression with both matrix and vector valued predictors
• \(\ell_{2,0}\)-norm based selection and estimation for multivariate generalized linear models
• Multivariate sparse Laplacian shrinkage for joint estimation of two graphical structures
• Nonparametric and high-dimensional functional graphical models
• Variable selection and collinearity processing for multivariate data via row-elastic-net regularization
• Adaptive estimation in multivariate response regression with hidden variables
• Sparse high-dimensional linear regression. Estimating squared error and a phase transition
• Large-scale multivariate sparse regression with applications to UK Biobank
• Communication-efficient distributed multi-task learning with matrix sparsity regularization
• Capturing between-tasks covariance and similarities using multivariate linear mixed models
• Consistent group selection with Bayesian high dimensional modeling
• A sequential approach to feature selection in high-dimensional additive models
• Prediction and estimation consistency of sparse multi-class penalized optimal scoring
• Adaptive estimation of the rank of the coefficient matrix in high-dimensional multivariate response regression models
• Recovery of simultaneous low rank and two-way sparse coefficient matrices, a nonconvex approach
• A consistent variable selection method in high-dimensional canonical discriminant analysis
• A two-stage sequential conditional selection approach to sparse high-dimensional multivariate regression models
• On model selection consistency of regularized M-estimators
• Structured regularization for conditional Gaussian graphical models
• Sparse conditional copula models for structured output regression
• Prototype selection for parameter estimation in complex models
• Structure estimation for discrete graphical models: generalized covariance matrices and their inverses
• Functional linear regression for functional response via sparse basis selection
• Sparse reduced-rank regression with covariance estimation
• An integrated precision matrix estimation for multivariate regression problems
• The benefit of group sparsity in group inference with de-biased scaled group Lasso
• A partially linear framework for massive heterogeneous data
• Simultaneous Variable and Covariance Selection With the Multivariate Spike-and-Slab LASSO
• Multivariate sparse group Lasso for the multivariate multiple linear regression with an arbitrary group structure
• The geometry of least squares in the 21st century
• ``Grouping strategies and thresholding for high dimensional linear models: discussion
• Lasso penalized model selection criteria for high-dimensional multivariate linear regression analysis
• Sparse regression and support recovery with \(\mathbb{L}_2\)-boosting algorithms
• Robust inference on average treatment effects with possibly more covariates than observations
• Modeling association between multivariate correlated outcomes and high-dimensional sparse covariates: the adaptive SVS method
• Regularized projection score estimation of treatment effects in high-dimensional quantile regression
• An Explicit Mean-Covariance Parameterization for Multivariate Response Linear Regression
• Asymptotic theory of _1-regularized PDE identification from a single noisy trajectory
• Feature selection by canonical correlation search in high-dimensional multiresponse models with complex group structures
• CLEAR: covariant least-square refitting with applications to image restoration
• Bayesian sparse partial least squares
• Union support recovery in multi-task learning
• Unsupervised learning of compositional sparse code for natural image representation
• Estimation of (near) low-rank matrices with noise and high-dimensional scaling
• Variable screening in multivariate linear regression with high-dimensional covariates
• Estimation of Low Rank High-Dimensional Multivariate Linear Models for Multi-Response Data
• Structured sparsity through convex optimization
• A selective review of group selection in high-dimensional models
• A unified framework for high-dimensional analysis of M-estimators with decomposable regularizers
• Sparse quadratic classification rules via linear dimension reduction
• Grouped variable selection with discrete optimization: computational and statistical perspectives
• Solution sets of three sparse optimization problems for multivariate regression
• A general framework for penalized mixed-effects multitask learning with applications on DNA methylation surrogate biomarkers creation
• Binned multinomial logistic regression for integrative cell-type annotation
• Optimal decorrelated score subsampling for generalized linear models with massive data
• Adaptive and robust multi-task learning
• An \(\ell_{2,0}\)-norm constrained matrix optimization via extended discrete first-order algorithms
• Estimation and inference in sparse multivariate regression and conditional Gaussian graphical models under an unbalanced distributed setting
• Oracle inequalities and optimal inference under group sparsity
• Scalable efficient reproducible multi-task learning via data splitting
• Non-convex penalized multitask regression using data depth-based penalties
• Fast optimization methods for high-dimensional row-sparse multivariate quantile linear regression
• Noise covariance estimation in multi-task high-dimensional linear models
• A review of Bayesian group selection approaches for linear regression models
• Distributed optimisation approach to least-squares solution of Sylvester equations
• Optimal Poisson subsampling decorrelated score for high-dimensional generalized linear models
• Interaction screening in high-dimensional multi-response regression via projected distance correlation
• Iterative sequential screening strategies for sparse recovery with computational advantages
• Robust multitask feature learning with adaptive Huber regressions
• Meta representation learning with contextual linear bandits
• Estimation of the error structure in multivariate response linear regression models
• Homogeneity estimation in multivariate generalized linear models
• Statistical inference for matrix-vector linear regression without debiasing under Kronecker covariance structure
• A model-free multivariate non-recursive feature elimination for feature selection on high-dimensional complex multiple response data
• Optimal Decorrelated Score Subsampling for High-Dimensional Generalized Linear Models Under Measurement Constraints
• Low-rank regularization in two-sided matrix regression
• Multiple regression for matrix and vector predictors: models, theory, algorithms, and beyond
• Tree-guided group lasso for multi-response regression with structured sparsity, with an application to eQTL mapping
• Hypothesis test in high dimensional multi-response linear models
• Double-robust and heterogeneity-aware distributed Poisson subsampling for misspecified measurement error models
• Multivariate regression modeling in integrative analysis via sparse regularization
• Factor Augmented Matrix Regression
• Penalized estimation in finite mixtures of multivariate regression models via the EM-PGM algorithm
• Tight conditions for consistency of variable selection in the context of high dimensionality
• Accuracy guaranties for \(\ell_{1}\) recovery of block-sparse signals
• Variable selection in high-dimensional sparse multiresponse linear regression models
• Variable selection in multivariate linear models for functional data via sparse regularization
• Generalized co-sparse factor regression
• Optimal variable selection in multi-group sparse discriminant analysis
• High-dimensional Ising model selection using \(\ell _{1}\)-regularized logistic regression
• Regularized multivariate regression for identifying master predictors with application to integrative genomics study of breast cancer

Describes a project that uses

Uses Software

• PDCO

This page was built for publication: Support union recovery in high-dimensional multivariate regression

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2429923)