Subset Selection with Shrinkage: Sparse Linear Modeling When the SNR Is Low

DOI10.1287/OPRE.2022.2276MaRDI QIDQ6202664zbMATH OpenOpenAlexFDO

Authors Rahul Mazumder, Peter Radchenko, Antoine Dedieu

Publication date 26 February 2024

Published in Operations Research (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1708.03288, https://hdl.handle.net/1721.1/144220

zbMATH Keywords

best subset selectionerror boundshigh-dimensional statisticsinteger programmingmachine learning and data scienceoracle inequalitiessignal-to-noise ratiosparse regression

Mathematics Subject Classification ID

Mathematical programming (90Cxx)

Abstract: We study a seemingly unexpected and relatively less understood overfitting aspect of a fundamental tool in sparse linear modeling - best subset selection, which minimizes the residual sum of squares subject to a constraint on the number of nonzero coefficients. While the best subset selection procedure is often perceived as the "gold standard" in sparse learning when the signal to noise ratio (SNR) is high, its predictive performance deteriorates when the SNR is low. In particular, it is outperformed by continuous shrinkage methods, such as ridge regression and the Lasso. We investigate the behavior of best subset selection in the high-noise regimes and propose an alternative approach based on a regularized version of the least-squares criterion. Our proposed estimators (a) mitigate, to a large extent, the poor predictive performance of best subset selection in the high-noise regimes; and (b) perform favorably, while generally delivering substantially sparser models, relative to the best predictive models available via ridge regression and the Lasso. We conduct an extensive theoretical analysis of the predictive properties of the proposed approach and provide justification for its superior predictive performance relative to best subset selection when the noise-level is high. Our estimators can be expressed as solutions to mixed integer second order conic optimization problems and, hence, are amenable to modern computational tools from mathematical optimization.

Recommendations

• Robust subset selection
• Better subset regression
• Best subset selection, persistence in high-dimensional statistical learning and optimization under l₁ constraint
• COMBSS: best subset selection via continuous optimization
• Near-ideal model selection by \(\ell _{1}\) minimization

Cited in

(12)

• HARFE: hard-ridge random feature expansion
• A General Framework for Identifying Hierarchical Interactions and Its Application to Genomics Data
• Best subset selection with shrinkage: sparse additive hazards regression with the grouping effect
• SRMD: sparse random mode decomposition
• An _0-constrained and _1-regularized estimator for graphical models
• Best-subset instrumental variable selection method using mixed integer optimization with applications to health-related quality of life and education-wage analyses
• Predicting census survey response rates with parsimonious additive models and structured interactions
• Group Selection and Shrinkage: Structured Sparsity for Semiparametric Additive Models
• Feature and functional form selection in additive models via mixed-integer optimization
• Optimal forecast reconciliation with time series selection
• Rank-one convexification for sparse regression
• Multi-task learning for sparsity pattern heterogeneity: statistical and computational perspectives

This page was built for publication: Subset Selection with Shrinkage: Sparse Linear Modeling When the SNR Is Low

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6202664)