Asymptotic properties of Lasso+mLS and Lasso+Ridge in sparse high-dimensional linear regression

DOI10.1214/14-EJS875zbMATH Open1281.62158arXiv1306.5505OpenAlexW2963705952MaRDI QIDQ389956FDOQ389956

Authors: Hanzhong Liu, Bin Yu 

Publication date: 22 January 2014

Published in: Electronic Journal of Statistics (Search for Journal in Brave)

Abstract: We study the asymptotic properties of Lasso+mLS and Lasso+Ridge under the sparse high-dimensional linear regression model: Lasso selecting predictors and then modified Least Squares (mLS) or Ridge estimating their coefficients. First, we propose a valid inference procedure for parameter estimation based on parametric residual bootstrap after Lasso+mLS and Lasso+Ridge. Second, we derive the asymptotic unbiasedness of Lasso+mLS and Lasso+Ridge. More specifically, we show that their biases decay at an exponential rate and they can achieve the oracle convergence rate of $s/n$ (where $s$ is the number of nonzero regression coefficients and $n$ is the sample size) for mean squared error (MSE). Third, we show that Lasso+mLS and Lasso+Ridge are asymptotically normal. They have an oracle property in the sense that they can select the true predictors with probability converging to 1 and the estimates of nonzero parameters have the same asymptotic normal distribution that they would have if the zero parameters were known in advance. In fact, our analysis is not limited to adopting Lasso in the selection stage, but is applicable to any other model selection criteria with exponentially decay rates of the probability of selecting wrong models.

Full work available at URL: https://arxiv.org/abs/1306.5505

Recommendations

• Adaptive Lasso for sparse high-dimensional regression models
• Least squares after model selection in high-dimensional sparse models
• Adaptive Lasso in high-dimensional settings
• The sparsity and bias of the LASSO selection in high-dimensional linear regression
• Asymptotic properties of lasso in high-dimensional partially linear models

zbMATH Keywords

asymptotic normalitysparsityasymptotic unbiasednessirrepresentable conditionLasso+Ridgeresidual bootstrap

Mathematics Subject Classification ID

Asymptotic properties of parametric estimators (62F12) Bootstrap, jackknife and other resampling methods (62F40) Ridge regression; shrinkage estimators (Lasso) (62J07)

Cites Work

• Title not available (Why is that?)
• A unified approach to model selection and sparse recovery using regularized least squares
• The Adaptive Lasso and Its Oracle Properties
• Least angle regression. (With discussion)
• Pathwise coordinate optimization
• Title not available (Why is that?)
• Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties
• Title not available (Why is that?)
• Lasso-type recovery of sparse representations for high-dimensional data
• Bootstrap methods: another look at the jackknife
• Simultaneous analysis of Lasso and Dantzig selector
• High-dimensional generalized linear models and the lasso
• The Dantzig selector: statistical estimation when \(p\) is much larger than \(n\). (With discussions and rejoinder).
• High-dimensional graphs and variable selection with the Lasso
• Title not available (Why is that?)
• Sharp Thresholds for High-Dimensional and Noisy Sparsity Recovery Using $\ell _{1}$-Constrained Quadratic Programming (Lasso)
• Regularization and Variable Selection Via the Elastic Net
• Ridge Regression: Biased Estimation for Nonorthogonal Problems
• A unified framework for high-dimensional analysis of \(M\)-estimators with decomposable regularizers
• Bootstrapping regression models
• Persistene in high-dimensional linear predictor-selection and the virtue of overparametrization
• Asymptotics for Lasso-type estimators.
• Least squares after model selection in high-dimensional sparse models
• Sparsity oracle inequalities for the Lasso
• \(\ell_1\)-penalized quantile regression in high-dimensional sparse models
• Bootstrapping Lasso estimators
• Adaptive Lasso for sparse high-dimensional regression models
• Rates of convergence of the adaptive LASSO estimators to the oracle distribution and higher order refinements by the bootstrap
• MODEL SELECTION AND INFERENCE: FACTS AND FICTION
• The sparsity and bias of the LASSO selection in high-dimensional linear regression
• Asymptotic expansions for the power of distributionfree tests in the two- sample problem
• Minimax Rates of Estimation for High-Dimensional Linear Regression Over $\ell_q$-Balls
• Relaxed Lasso
• Stable recovery of sparse overcomplete representations in the presence of noise
• Some asymptotic theory for the bootstrap
• Contributions of empirical and quantile processes to the asymptotic theory of goodness-of-fit tests. (With comments)
• Asymptotic properties of bridge estimators in sparse high-dimensional regression models
• On the distribution of the adaptive LASSO estimator
• A perturbation method for inference on regularized regression estimates
• Recovery of Exact Sparse Representations in the Presence of Bounded Noise
• Greed is Good: Algorithmic Results for Sparse Approximation
• Honest variable selection in linear and logistic regression models via \(\ell _{1}\) and \(\ell _{1}+\ell _{2}\) penalization
• Sup-norm convergence rate and sign concentration property of Lasso and Dantzig estimators
• Title not available (Why is that?)
• Model selection consistency of Dantzig selector

Cited In (26)

• High-dimensional inference: confidence intervals, \(p\)-values and R-software \texttt{hdi}
• Hierarchical inference for genome-wide association studies: a view on methodology with software
• Estimation in the presence of heteroskedasticity of unknown form: a Lasso-based approach
• A two-stage bridge estimator for regression models with endogeneity based on control function method
• Markov Neighborhood Regression for High-Dimensional Inference
• Projection-based Inference for High-dimensional Linear Models
• Change points detection and parameter estimation for multivariate time series
• Statistical Inference for High-Dimensional Models via Recursive Online-Score Estimation
• Post-model-selection inference in linear regression models: an integrated review
• Ridge regression revisited: debiasing, thresholding and bootstrap
• Exploiting Disagreement Between High-Dimensional Variable Selectors for Uncertainty Visualization
• Random weighting in LASSO regression
• Lasso meets horseshoe: a survey
• Asymptotically faster estimation of high-dimensional additive models using subspace learning
• Lasso in Infinite dimension: application to variable selection in functional multivariate linear regression
• Bootstrap confidence regions based on M-estimators under nonstandard conditions
• Title not available (Why is that?)
• Kernel meets sieve: post-regularization confidence bands for sparse additive model
• Confidence intervals for parameters in high-dimensional sparse vector autoregression
• Rejoinder on: ``High-dimensional simultaneous inference with the bootstrap
• Comments on: ``High-dimensional simultaneous inference with the bootstrap
• The sparsity and bias of the LASSO selection in high-dimensional linear regression
• High-dimensional simultaneous inference with the bootstrap
• Beyond support in two-stage variable selection
• Rates of convergence of the adaptive elastic net and the post-selection procedure in ultra-high dimensional sparse models
• A bootstrap Lasso+partial ridge method to construct confidence intervals for parameters in high-dimensional sparse linear models

Uses Software

• bootstrap
• Bolasso
• bootlib

This page was built for publication: Asymptotic properties of Lasso+mLS and Lasso+Ridge in sparse high-dimensional linear regression

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q389956)