Least squares after model selection in high-dimensional sparse models
From MaRDI portal
Abstract: In this article we study post-model selection estimators that apply ordinary least squares (OLS) to the model selected by first-step penalized estimators, typically Lasso. It is well known that Lasso can estimate the nonparametric regression function at nearly the oracle rate, and is thus hard to improve upon. We show that the OLS post-Lasso estimator performs at least as well as Lasso in terms of the rate of convergence, and has the advantage of a smaller bias. Remarkably, this performance occurs even if the Lasso-based model selection "fails" in the sense of missing some components of the "true" regression model. By the "true" model, we mean the best s-dimensional approximation to the nonparametric regression function chosen by the oracle. Furthermore, OLS post-Lasso estimator can perform strictly better than Lasso, in the sense of a strictly faster rate of convergence, if the Lasso-based model selection correctly includes all components of the "true" model as a subset and also achieves sufficient sparsity. In the extreme case, when Lasso perfectly selects the "true" model, the OLS post-Lasso estimator becomes the oracle estimator. An important ingredient in our analysis is a new sparsity bound on the dimension of the model selected by Lasso, which guarantees that this dimension is at most of the same order as the dimension of the "true" model. Our rate results are nonasymptotic and hold in both parametric and nonparametric models. Moreover, our analysis is not limited to the Lasso estimator acting as a selector in the first step, but also applies to any other estimator, for example, various forms of thresholded Lasso, with good rates and good sparsity properties. Our analysis covers both traditional thresholding and a new practical, data-driven thresholding scheme that induces additional sparsity subject to maintaining a certain goodness of fit. The latter scheme has theoretical guarantees similar to those of Lasso or OLS post-Lasso, but it dominates those procedures as well as traditional thresholding in a wide variety of experiments.
Recommendations
Cites work
- scientific article; zbMATH DE number 5957408 (Why is no real title available?)
- scientific article; zbMATH DE number 845714 (Why is no real title available?)
- scientific article; zbMATH DE number 1420699 (Why is no real title available?)
- Aggregation and Sparsity Via ℓ1 Penalized Least Squares
- Aggregation for Gaussian regression
- All of Nonparametric Statistics
- High-dimensional generalized linear models and the lasso
- Introduction to nonparametric estimation
- Lasso-type recovery of sparse representations for high-dimensional data
- Least squares after model selection in high-dimensional sparse models
- Nonparametric curve estimation. Methods, theory, and applications
- On sparse reconstruction from Fourier and Gaussian measurements
- Oracle inequalities and optimal inference under group sparsity
- Sharp Thresholds for High-Dimensional and Noisy Sparsity Recovery Using $\ell _{1}$-Constrained Quadratic Programming (Lasso)
- Simultaneous analysis of Lasso and Dantzig selector
- Sparse recovery under matrix uncertainty
- Sparsity in penalized empirical risk minimization
- Sparsity oracle inequalities for the Lasso
- Sup-norm convergence rate and sign concentration property of Lasso and Dantzig estimators
- The Dantzig selector: statistical estimation when \(p\) is much larger than \(n\). (With discussions and rejoinder).
- The sparsity and bias of the LASSO selection in high-dimensional linear regression
- Weak convergence and empirical processes. With applications to statistics
- \(\ell_1\)-penalized quantile regression in high-dimensional sparse models
Cited in
(only showing first 100 items - show all)- Multiple imputation and selection of ordinal level 2 predictors in multilevel models: an analysis of the relationship between student ratings and teacher practices and attitudes
- De-biasing the Lasso with degrees-of-freedom adjustment
- Regularization methods for high-dimensional sparse control function models
- Does data splitting improve prediction?
- Generalized M-estimators for high-dimensional Tobit I models
- A comment on Hansen's risk of James-Stein and Lasso shrinkage
- Regularized covariance matrix estimation in high dimensional approximate factor models
- Nested model averaging on solution path for high-dimensional linear regression
- MuSP: a multistep screening procedure for sparse recovery
- Additive model selection
- Beyond support in two-stage variable selection
- CLEAR: covariant least-square refitting with applications to image restoration
- On the use of the Lasso for instrumental variables estimation with some invalid instruments
- An integrated precision matrix estimation for multivariate regression problems
- Debiased Inference on Treatment Effect in a High-Dimensional Model
- Information criteria bias correction for group selection
- Rates of convergence of the adaptive elastic net and the post-selection procedure in ultra-high dimensional sparse models
- Lasso-based variable selection methods in text regression: the case of short texts
- scientific article; zbMATH DE number 7306878 (Why is no real title available?)
- ROCKET: robust confidence intervals via Kendall's tau for transelliptical graphical models
- Asymptotic properties of Lasso+mLS and Lasso+Ridge in sparse high-dimensional linear regression
- On cross-validated Lasso in high dimensions
- Statistical inference in sparse high-dimensional additive models
- Inference for High-Dimensional Exchangeable Arrays
- Shrinkage estimation of dynamic panel data models with interactive fixed effects
- Lassoing the determinants of retirement
- Recovery of partly sparse and dense signals
- Prediction with a flexible finite mixture-of-regressions
- Combined \(\ell_1\) and greedy \(\ell_0\) penalized least squares for linear model selection
- The effect of regularization in portfolio selection problems
- Transaction cost analytics for corporate bonds
- Estimation in the presence of heteroskedasticity of unknown form: a Lasso-based approach
- Sorted concave penalized regression
- Least squares after model selection in high-dimensional sparse models
- A two-stage bridge estimator for regression models with endogeneity based on control function method
- Estimation of Linear Functionals in High-Dimensional Linear Models: From Sparsity to Nonsparsity
- Robust measurement via a fused latent and graphical item response theory model
- UNIFORM-IN-SUBMODEL BOUNDS FOR LINEAR REGRESSION IN A MODEL-FREE FRAMEWORK
- Complete subset regressions with large-dimensional sets of predictors
- Projected spline estimation of the nonparametric function in high-dimensional partially linear models for massive data
- A latent class Cox model for heterogeneous time-to-event data
- Inference for high-dimensional instrumental variables regression
- The risk of James-Stein and Lasso shrinkage
- Parametric and semiparametric reduced-rank regression with flexible sparsity
- Time-varying forecast combination for high-dimensional data
- Automated estimation of vector error correction models
- Time-dependent Poisson reduced rank models for political text data analysis
- Lower and upper bound estimates of inequality of opportunity for emerging economies
- Uniformly valid post-regularization confidence regions for many functional parameters in z-estimation framework
- Simultaneous inference for pairwise graphical models with generalized score matching
- Debiasing the Lasso: optimal sample size for Gaussian designs
- Gaussian approximations and multiplier bootstrap for maxima of sums of high-dimensional random vectors
- Using Machine Learning Methods to Support Causal Inference in Econometrics
- Variable selection and prediction with incomplete high-dimensional data
- Endogeneity in high dimensions
- Optimal model averaging for divergent-dimensional Poisson regressions
- Necessary and sufficient conditions for variable selection consistency of the Lasso in high dimensions
- Communication-efficient estimation of high-dimensional quantile regression
- I-LAMM for sparse learning: simultaneous control of algorithmic complexity and statistical error
- Recent advances in statistical methodologies in evaluating program for high-dimensional data
- Calibrating doubly-robust estimators with unbalanced treatment assignment
- Optimal bounds for aggregation of affine estimators
- Multiple structural breaks in cointegrating regressions: a model selection approach
- On estimation of the diagonal elements of a sparse precision matrix
- Reconciling model-X and doubly robust approaches to conditional independence testing
- SONIC: social network analysis with influencers and communities
- Block-based refitting in \(\ell_{12}\) sparse regularization
- A Critical Review of LASSO and Its Derivatives for Variable Selection Under Dependence Among Covariates
- A dynamic screening algorithm for hierarchical binary marketing data
- Inference in Additively Separable Models With a High-Dimensional Set of Conditioning Variables
- Statistical inference for model parameters in stochastic gradient descent
- Sparse linear models and \(l_1\)-regularized 2SLS with high-dimensional endogenous regressors and instruments
- Inference on heterogeneous treatment effects in high‐dimensional dynamic panels under weak dependence
- On Lasso refitting strategies
- Diffusion Indexes With Sparse Loadings
- Random weighting in LASSO regression
- Inference for biased transformation models
- Inference for low-rank models
- High-dimensional variable selection via low-dimensional adaptive learning
- Directed graphs and variable selection in large vector autoregressive models
- LASSO for Stochastic Frontier Models with Many Efficient Firms
- When are Google Data Useful to Nowcast GDP? An Approach via Preselection and Shrinkage
- Assessing the robustness of sisVIVE in a Mendelian randomization study to estimate the causal effect of body mass index on income using multiple SNPs from understanding society
- Inference in Approximately Sparse Correlated Random Effects Probit Models With Panel Data
- Propensity score weighting for causal subgroup analysis
- Model selection criteria for a linear model to solve discrete ill-posed problems on the basis of singular decomposition and random projection
- scientific article; zbMATH DE number 7415101 (Why is no real title available?)
- A projection-based conditional dependence measure with applications to high-dimensional undirected graphical models
- Control variate selection for Monte Carlo integration
- An alternative to synthetic control for models with many covariates under sparsity
- Robust analysis of cancer heterogeneity for high-dimensional data
- Fast rates of minimum error entropy with heavy-tailed noise
- Nonconvex penalized reduced rank regression and its oracle properties in high dimensions
- Sequential change point detection for high‐dimensional data using nonconvex penalized quantile regression
- In defense of the indefensible: a very naïve approach to high-dimensional inference
- Constrained mix sparse optimization via hard thresholding pursuit
- Increasing the efficiency of randomized trial estimates via linear adjustment for a prognostic score
- A penalized approach to covariate selection through quantile regression coefficient models
- On asymptotically optimal confidence regions and tests for high-dimensional models
- BAYESIAN DYNAMIC VARIABLE SELECTION IN HIGH DIMENSIONS
This page was built for publication: Least squares after model selection in high-dimensional sparse models
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q1952433)
