Projective inference in high-dimensional problems: prediction and feature selection
From MaRDI portal
Abstract: This paper discusses predictive inference and feature selection for generalized linear models with scarce but high-dimensional data. We argue that in many cases one can benefit from a decision theoretically justified two-stage approach: first, construct a possibly non-sparse model that predicts well, and then find a minimal subset of features that characterize the predictions. The model built in the first step is referred to as the emph{reference model} and the operation during the latter step as predictive emph{projection}. The key characteristic of this approach is that it finds an excellent tradeoff between sparsity and predictive accuracy, and the gain comes from utilizing all available information including prior and that coming from the left out features. We review several methods that follow this principle and provide novel methodological contributions. We present a new projection technique that unifies two existing techniques and is both accurate and fast to compute. We also propose a way of evaluating the feature selection process using fast leave-one-out cross-validation that allows for easy and intuitive model size selection. Furthermore, we prove a theorem that helps to understand the conditions under which the projective approach could be beneficial. The benefits are illustrated via several simulated and real world examples.
The paper discusses predictive inference and feature selection for generalised linear models, when the data is highly dimensional but with a low number of samples, e.g. as in the case of microarray records. The article demonstrates that a two-step approach consisting of a reference model and a procedure for finding a minimal subset of features characteristic of the predictions is beneficial. Several techniques in this category are reviewed. A new clustered projection approach that is a combination of two other methods and has both speed and good accuracy is proposed. A novel LOO cross-validation method for evaluating the feature selection process is put forward. Also, a theorem for the conditions under which the projective approach is advantageous is proved. An R package is provided, where all the discussed methods are implemented. Several simulation studies performed and the application on five microarray data sets demonstrate also from the practical perspective the effectiveness of the approach.
Recommendations
- Projection-based Inference for High-dimensional Linear Models
- Targeted random projection for prediction from high-dimensional features
- Ultrahigh dimensional feature screening via projection
- A Computational Perspective on Projection Pursuit in High Dimensions: Feasible or Infeasible Feature Extraction
- scientific article; zbMATH DE number 2042277
- Some perspectives on inference in high dimensions
- On the conditional distributions of low-dimensional projections from high-dimensional data
- Projective resampling estimation of informative predictor subspace for multivariate regression
- scientific article; zbMATH DE number 2043421
- High-dimensional variable selection with sparse random projections: measurement sparsity and statistical efficiency
Cites work
- 10.1162/153244303322753715
- ``Preconditioning for feature selection and regression in high-dimensional problems
- A decision-theoretic approach for model interpretability in Bayesian framework
- A study of error variance estimation in Lasso regression
- A survey of Bayesian predictive methods for model assessment, selection and comparison
- Bayesian data analysis.
- Bayesian Model Averaging for Linear Regression Models
- Bayesian model selection in high-dimensional settings
- Bayesian projection approaches to variable selection in generalized linear models
- Bayesian variable selection with shrinking and diffusing priors
- Better Subset Regression Using the Nonnegative Garrote
- Comparison of Bayesian predictive methods for model selection
- Decoupling shrinkage and selection in Bayesian linear models: a posterior summary perspective
- Dirichlet-Laplace priors for optimal shrinkage
- scientific article; zbMATH DE number 47310 (Why is no real title available?)
- scientific article; zbMATH DE number 578421 (Why is no real title available?)
- scientific article; zbMATH DE number 845714 (Why is no real title available?)
- scientific article; zbMATH DE number 6438182 (Why is no real title available?)
- scientific article; zbMATH DE number 3249515 (Why is no real title available?)
- Implicitly adaptive importance sampling
- Large-scale inference. Empirical Bayes methods for estimation, testing, and prediction
- Least angle regression. (With discussion)
- Model choice in generalised linear models: a Bayesian approach via Kullback-Leibler projections
- Needles and straw in a haystack: posterior concentration for possibly sparse sequences
- Needles and straw in haystacks: Empirical Bayes estimates of possibly sparse sequences
- On over-fitting in model selection and subsequent selection bias in performance evaluation
- Optimal predictive model selection.
- Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC
- Prediction by Supervised Principal Components
- Regression modeling strategies. With applications to linear models, logistic regression, and survival analysis
- Regularization and Variable Selection Via the Elastic Net
- Relaxed Lasso
- Scalable Importance Tempering and Bayesian Variable Selection
- Selection bias in gene extraction on the basis of microarray gene-expression data
- Sparsity information and regularization in the horseshoe and other shrinkage priors
- Spike and slab variable selection: frequentist and Bayesian strategies
- Sure independence screening for ultrahigh dimensional feature space. With discussion and authors' reply
- The Adaptive Lasso and Its Oracle Properties
- The Dantzig selector: statistical estimation when \(p\) is much larger than \(n\). (With discussions and rejoinder).
- The horseshoe estimator for sparse signals
- The horseshoe estimator: posterior concentration around nearly black vectors
- The horseshoe+ estimator of ultra-sparse signals
- The predictive Lasso
- Using stacking to average Bayesian predictive distributions (with discussion)
- Variable selection in qualitative models via an entropic explanatory power
- Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties
Cited in
(18)- Projective resampling estimation of informative predictor subspace for multivariate regression
- A decision-theoretic approach for model interpretability in Bayesian framework
- Targeted random projection for prediction from high-dimensional features
- Bayesian estimation of subset threshold autoregressions: short-term forecasting of traffic occupancy
- Using reference models in variable selection
- Fast, Optimal, and Targeted Predictions Using Parameterized Decision Analysis
- A fully Bayesian sparse polynomial chaos expansion approach with joint priors on the coefficients and global selection of terms
- Intuitive joint priors for Bayesian linear multilevel models: the R2D2M2 prior
- Structured Shrinkage Priors
- Efficient estimation and correction of selection-induced bias with order statistics
- Lower-dimensional posterior density and cluster summaries for overparameterized Bayesian models
- Expressing and visualizing model uncertainty in Bayesian variable selection using Cartesian credible sets
- Uncertainty quantification and propagation in surrogate-based Bayesian inference
- Projection predictive variable selection for discrete response families with finite support
- Advances in projection predictive inference
- Flexible Bayesian MIDAS: Time-Variation, Group-Shrinkage and Sparsity
- The ARR2 prior: flexible predictive prior definition for Bayesian auto-regressions
- Generalized decomposition priors on R2
Describes a project that uses
Uses Software
This page was built for publication: Projective inference in high-dimensional problems: prediction and feature selection
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2188473)
