On cross-validated Lasso in high dimensions

DOI10.1214/20-AOS2000MaRDI QIDQ820794zbMATH OpenOpenAlexFDO

Authors Denis Chetverikov, Zhipeng Liao, Victor Chernozhukov

Publication date 28 September 2021

Published in The Annals of Statistics (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1605.02214

zbMATH Keywords

cross-validationhigh-dimensional modelsnonasymptotic boundssparsity

Mathematics Subject Classification ID

Nonparametric estimation (62G05) Asymptotic properties of nonparametric inference (62G20) Ridge regression; shrinkage estimators (Lasso) (62J07)

Abstract: In this paper, we derive non-asymptotic error bounds for the Lasso estimator when the penalty parameter for the estimator is chosen using $K$-fold cross-validation. Our bounds imply that the cross-validated Lasso estimator has nearly optimal rates of convergence in the prediction, $L^2$, and $L^1$ norms. For example, we show that in the model with the Gaussian noise and under fairly general assumptions on the candidate set of values of the penalty parameter, the estimation error of the cross-validated Lasso estimator converges to zero in the prediction norm with the $sqrtslogp/nimessqrtlog(pn)$ rate, where $n$ is the sample size of available data, $p$ is the number of covariates, and $s$ is the number of non-zero coefficients in the model. Thus, the cross-validated Lasso estimator achieves the fastest possible rate of convergence in the prediction norm up to a small logarithmic factor $sqrtlog(pn)$, and similar conclusions apply for the convergence rate both in $L^2$ and in $L^1$ norms. Importantly, our results cover the case when $p$ is (potentially much) larger than $n$ and also allow for the case of non-Gaussian noise. Our paper therefore serves as a justification for the widely spread practice of using cross-validation as a method to choose the penalty parameter for the Lasso estimator.

Recommendations

• High-dimensional generalized linear models and the lasso
• Cross validation in Lasso and its acceleration
• Comparison of lasso type estimators for high-dimensional data
• Stabilizing the Lasso against cross-validation variability
• The sparsity and bias of the LASSO selection in high-dimensional linear regression
• A study on tuning parameter selection for the high-dimensional lasso
• Lasso penalized model selection criteria for high-dimensional multivariate linear regression analysis
• Adaptive Lasso in high-dimensional settings
• Necessary and sufficient conditions for variable selection consistency of the Lasso in high dimensions
• Shrinkage and LASSO strategies in high-dimensional heteroscedastic models

Cites work

• \(\ell_1\)-penalized quantile regression in high-dimensional sparse models
• A survey of cross-validation procedures for model selection
• Asymptotic optimality for \(C_ p\), \(C_ L\), cross-validation and generalized cross-validation: Discrete index set
• Degrees of freedom in lasso problems
• Estimation and testing under sparsity. École d'Été de Probabilités de Saint-Flour XLV -- 2015
• Exponential screening and optimal rates of sparse estimation
• scientific article; zbMATH DE number 6388313 (Why is no real title available?)
• scientific article; zbMATH DE number 845714 (Why is no real title available?)
• scientific article; zbMATH DE number 6438182 (Why is no real title available?)
• Least squares after model selection in high-dimensional sparse models
• Leave-one-out cross-validation is risk consistent for Lasso
• Model selection in nonparametric regression
• Nearly unbiased variable selection under minimax concave penalty
• On the ``degrees of freedom of the lasso
• Oracle inequalities for cross-validation type procedures
• Regularization and the small-ball method. I: Sparse recovery
• Risk consistency of cross-validation with Lasso-type procedures
• Simultaneous analysis of Lasso and Dantzig selector
• Some new asymptotic theory for least squares series: pointwise and uniform results
• Sparse matrix inversion with scaled Lasso
• Statistics for high-dimensional data. Methods, theory and applications.
• The Lasso problem and uniqueness
• The sparsity and bias of the LASSO selection in high-dimensional linear regression
• Weak convergence and empirical processes. With applications to statistics

Cited in

(39)

• Stabilizing the Lasso against cross-validation variability
• The distribution of the Lasso: uniform control over sparse balls and adaptive parameter tuning
• Aggregated hold out for sparse linear regression with a robust loss function
• High-dimensional regression with potential prior information on variable importance
• Regularization methods for high-dimensional sparse control function models
• Leave-one-out cross-validation is risk consistent for Lasso
• Shrinkage and LASSO strategies in high-dimensional heteroscedastic models
• A practical scheme and fast algorithm to tune the Lasso with optimality guarantees
• Cross validation in Lasso and its acceleration
• Oracle inequalities for multi-fold cross validation
• The cross-validated adaptive epsilon-net estimator
• Accelerating cross-validation in multinomial logistic regression with \(\ell_1\)-regularization
• Using Machine Learning Methods to Support Causal Inference in Econometrics
• On the prediction performance of the Lasso
• scientific article; zbMATH DE number 7626791 (Why is no real title available?)
• Cross validation in sparse linear regression with piecewise continuous nonconvex penalties and its acceleration
• An analysis of the cost of hyperparameter selection via split-sample validation, with applications to penalized regression
• Risk consistency of cross-validation with Lasso-type procedures
• Targeted estimation of state occupation probabilities for the non‐Markov illness‐death model
• Tuning parameter selection for penalized estimation via R^2
• Analyzing evidence-based falls prevention data with significant missing information using variable selection after multiple imputation
• The Lasso with general Gaussian designs with applications to hypothesis testing
• -Penalized Pairwise Difference Estimation for a High-Dimensional Censored Regression Model
• Confidence intervals for intentionally biased estimators
• A nonparametric mixed-effects mixture model for patterns of clinical measurements associated with COVID-19
• Cross-validation for change-point regression: pitfalls and solutions
• Cross-validation on extreme regions
• Estimation and inference in high-dimensional panel data models with interactive fixed effects
• Doubly-robust inference for conditional average treatment effects with high-dimensional controls
• Moran's I Lasso for models with spatially correlated data
• Review of simulation studies evaluating imputation methods in high-dimensional datasets
• F statistics for high-dimensional inference of linear model
• Tuning parameter selection for the adaptive nuclear norm regularized trace regression
• One component partial least squares, high dimensional regression, data splitting, and the multitude of models
• The distributionally robust prediction error of the \(\sqrt{\mathrm{LASSO}}\) and related estimators
• High-dimensional conditional factor model
• Small Tuning Parameter Selection for the Debiased Lasso
• Estimating individual responses when tomorrow matters
• Double machine learning for a partially linear model with endogenous treatments and multivariate sample selection

This page was built for publication: On cross-validated Lasso in high dimensions

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q820794)