``Preconditioning for feature selection and regression in high-dimensional problems

DOI10.1214/009053607000000578zbMATH Open1142.62022arXivmath/0703858OpenAlexW1997840761WikidataQ60691592 ScholiaQ60691592MaRDI QIDQ939656FDOQ939656

Authors: Debashis Paul, Eric Bair, Robert Tibshirani, Trevor Hastie 

Publication date: 28 August 2008

Published in: The Annals of Statistics (Search for Journal in Brave)

Abstract: We consider regression problems where the number of predictors greatly exceeds the number of observations. We propose a method for variable selection that first estimates the regression function, yielding a "pre-conditioned" response variable. The primary method used for this initial regression is supervised principal components. Then we apply a standard procedure such as forward stepwise selection or the LASSO to the pre-conditioned response variable. In a number of simulated and real data examples, this two-step procedure outperforms forward stepwise selection or the usual LASSO (applied directly to the raw outcome). We also show that under a certain Gaussian latent variable model, application of the LASSO to the pre-conditioned response variable is consistent as the number of predictors and observations increases. Moreover, when the observational noise is rather large, the suggested procedure can give a more accurate estimate than LASSO. We illustrate our method on some real problems, including survival analysis with microarray data.

Full work available at URL: https://arxiv.org/abs/math/0703858

Recommendations

• Prediction by Supervised Principal Components
• Preselection in Lasso-type analysis for ultra-high dimensional genomic exploration
• Sure independence screening for ultrahigh dimensional feature space. With discussion and authors' reply
• High-dimensional variable selection
• Factor models and variable selection in high-dimensional regression analysis

zbMATH Keywords

lassomodel selectionprediction error

Mathematics Subject Classification ID

Nonparametric regression and quantile regression (62G08) Asymptotic properties of nonparametric inference (62G20) Factor analysis and principal components; correspondence analysis (62H25) Ridge regression; shrinkage estimators (Lasso) (62J07)

Cites Work

• The Adaptive Lasso and Its Oracle Properties
• Least angle regression. (With discussion)
• Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties
• Title not available (Why is that?)
• High-dimensional graphs and variable selection with the Lasso
• Title not available (Why is that?)
• Prediction by Supervised Principal Components
• Asymptotics for Lasso-type estimators.
• Title not available (Why is that?)
• Optimally sparse representation in general (nonorthogonal) dictionaries via ℓ ¹ minimization
• Nonconcave penalized likelihood with a diverging number of parameters.
• Adaptive Model Selection

Cited In (30)

• Non-crossing large-margin probability estimation and its application to robust SVM via pre\-condi\-tion\-ing
• Feature selection: from the past to the future
• Projective inference in high-dimensional problems: prediction and feature selection
• Preselection in Lasso-type analysis for ultra-high dimensional genomic exploration
• Canonical thresholding for nonsparse high-dimensional linear regression
• On data preconditioning for regularized loss minimization
• Noisy matrix completion: understanding statistical guarantees for convex relaxation via nonconvex optimization
• Estimation of a regression function corresponding to latent~variables
• Variable selection for kernel classification
• High-dimensional Cox regression analysis in genetic studies with censored survival outcomes
• A variable selection proposal for multiple linear regression analysis
• Robust high-dimensional factor models with applications to statistical machine learning
• Preconditioning the Lasso for sign consistency
• Augmented sparse reconstruction of protein signaling networks
• Optimal EMG placement for a robotic prosthesis controller with sequential, adaptive functional estimation (SAFE)
• Bayesian projection approaches to variable selection in generalized linear models
• Statistical and knowledge supported visualization of multivariate data
• Nonparametric estimation of a latent variable model
• Adaptive index models for marker-based risk stratification
• Sure Independence Screening for Ultrahigh Dimensional Feature Space
• Tournament screening cum EBIC for feature selection with high-dimensional feature spaces
• High-dimensional model recovery from random sketched data by exploring intrinsic sparsity
• Compressed and Penalized Linear Regression
• Prediction by Supervised Principal Components
• Using reference models in variable selection
• Addressing the implementation challenge of risk prediction model due to missing risk factors: the submodel approximation approach
• Title not available (Why is that?)
• Use of pretransformation to cope with extreme values in important candidate features
• Preconditioning for classical relationships: a note relating ridge regression and OLS \(p\)-values to preconditioned \textit{sparse} penalized regression
• Dimensionality reduction by feature clustering for regression problems

This page was built for publication: ``Preconditioning for feature selection and regression in high-dimensional problems

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q939656)