Projective inference in high-dimensional problems: prediction and feature selection
From MaRDI portal
Publication:2188473
Abstract: This paper discusses predictive inference and feature selection for generalized linear models with scarce but high-dimensional data. We argue that in many cases one can benefit from a decision theoretically justified two-stage approach: first, construct a possibly non-sparse model that predicts well, and then find a minimal subset of features that characterize the predictions. The model built in the first step is referred to as the emph{reference model} and the operation during the latter step as predictive emph{projection}. The key characteristic of this approach is that it finds an excellent tradeoff between sparsity and predictive accuracy, and the gain comes from utilizing all available information including prior and that coming from the left out features. We review several methods that follow this principle and provide novel methodological contributions. We present a new projection technique that unifies two existing techniques and is both accurate and fast to compute. We also propose a way of evaluating the feature selection process using fast leave-one-out cross-validation that allows for easy and intuitive model size selection. Furthermore, we prove a theorem that helps to understand the conditions under which the projective approach could be beneficial. The benefits are illustrated via several simulated and real world examples.
Recommendations
- Projection-based Inference for High-dimensional Linear Models
- Targeted random projection for prediction from high-dimensional features
- Ultrahigh dimensional feature screening via projection
- A Computational Perspective on Projection Pursuit in High Dimensions: Feasible or Infeasible Feature Extraction
- scientific article; zbMATH DE number 2042277
- Some perspectives on inference in high dimensions
- On the conditional distributions of low-dimensional projections from high-dimensional data
- Projective resampling estimation of informative predictor subspace for multivariate regression
- scientific article; zbMATH DE number 2043421
- High-dimensional variable selection with sparse random projections: measurement sparsity and statistical efficiency
Cites work
- scientific article; zbMATH DE number 47310 (Why is no real title available?)
- scientific article; zbMATH DE number 578421 (Why is no real title available?)
- scientific article; zbMATH DE number 845714 (Why is no real title available?)
- scientific article; zbMATH DE number 6438182 (Why is no real title available?)
- scientific article; zbMATH DE number 3249515 (Why is no real title available?)
- 10.1162/153244303322753715
- A decision-theoretic approach for model interpretability in Bayesian framework
- A study of error variance estimation in Lasso regression
- A survey of Bayesian predictive methods for model assessment, selection and comparison
- Bayesian Model Averaging for Linear Regression Models
- Bayesian data analysis.
- Bayesian model selection in high-dimensional settings
- Bayesian projection approaches to variable selection in generalized linear models
- Bayesian variable selection with shrinking and diffusing priors
- Better Subset Regression Using the Nonnegative Garrote
- Comparison of Bayesian predictive methods for model selection
- Decoupling shrinkage and selection in Bayesian linear models: a posterior summary perspective
- Dirichlet-Laplace priors for optimal shrinkage
- Implicitly adaptive importance sampling
- Large-scale inference. Empirical Bayes methods for estimation, testing, and prediction
- Least angle regression. (With discussion)
- Model choice in generalised linear models: a Bayesian approach via Kullback-Leibler projections
- Needles and straw in a haystack: posterior concentration for possibly sparse sequences
- Needles and straw in haystacks: Empirical Bayes estimates of possibly sparse sequences
- On over-fitting in model selection and subsequent selection bias in performance evaluation
- Optimal predictive model selection.
- Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC
- Prediction by Supervised Principal Components
- Regression modeling strategies. With applications to linear models, logistic regression, and survival analysis
- Regularization and Variable Selection Via the Elastic Net
- Relaxed Lasso
- Scalable Importance Tempering and Bayesian Variable Selection
- Selection bias in gene extraction on the basis of microarray gene-expression data
- Sparsity information and regularization in the horseshoe and other shrinkage priors
- Spike and slab variable selection: frequentist and Bayesian strategies
- Sure independence screening for ultrahigh dimensional feature space. With discussion and authors' reply
- The Adaptive Lasso and Its Oracle Properties
- The Dantzig selector: statistical estimation when \(p\) is much larger than \(n\). (With discussions and rejoinder).
- The horseshoe estimator for sparse signals
- The horseshoe estimator: posterior concentration around nearly black vectors
- The horseshoe+ estimator of ultra-sparse signals
- The predictive Lasso
- Using stacking to average Bayesian predictive distributions (with discussion)
- Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties
- Variable selection in qualitative models via an entropic explanatory power
- ``Preconditioning for feature selection and regression in high-dimensional problems
Cited in
(10)- Structured Shrinkage Priors
- Efficient estimation and correction of selection-induced bias with order statistics
- Fast, Optimal, and Targeted Predictions Using Parameterized Decision Analysis
- Using reference models in variable selection
- A fully Bayesian sparse polynomial chaos expansion approach with joint priors on the coefficients and global selection of terms
- Targeted random projection for prediction from high-dimensional features
- Bayesian estimation of subset threshold autoregressions: short-term forecasting of traffic occupancy
- Projective resampling estimation of informative predictor subspace for multivariate regression
- Intuitive joint priors for Bayesian linear multilevel models: the R2D2M2 prior
- A decision-theoretic approach for model interpretability in Bayesian framework
Describes a project that uses
Uses Software
This page was built for publication: Projective inference in high-dimensional problems: prediction and feature selection
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2188473)
