Projective inference in high-dimensional problems: prediction and feature selection
From MaRDI portal
Publication:2188473
DOI10.1214/20-EJS1711zbMATH Open1476.62058arXiv1810.02406OpenAlexW2894867725MaRDI QIDQ2188473FDOQ2188473
Authors: Juho Piironen, Markus Paasiniemi, Aki Vehtari 
Publication date: 11 June 2020
Published in: Electronic Journal of Statistics (Search for Journal in Brave)
Abstract: This paper discusses predictive inference and feature selection for generalized linear models with scarce but high-dimensional data. We argue that in many cases one can benefit from a decision theoretically justified two-stage approach: first, construct a possibly non-sparse model that predicts well, and then find a minimal subset of features that characterize the predictions. The model built in the first step is referred to as the emph{reference model} and the operation during the latter step as predictive emph{projection}. The key characteristic of this approach is that it finds an excellent tradeoff between sparsity and predictive accuracy, and the gain comes from utilizing all available information including prior and that coming from the left out features. We review several methods that follow this principle and provide novel methodological contributions. We present a new projection technique that unifies two existing techniques and is both accurate and fast to compute. We also propose a way of evaluating the feature selection process using fast leave-one-out cross-validation that allows for easy and intuitive model size selection. Furthermore, we prove a theorem that helps to understand the conditions under which the projective approach could be beneficial. The benefits are illustrated via several simulated and real world examples.
Full work available at URL: https://arxiv.org/abs/1810.02406
Recommendations
- Projection-based Inference for High-dimensional Linear Models
- Targeted random projection for prediction from high-dimensional features
- Ultrahigh dimensional feature screening via projection
- A Computational Perspective on Projection Pursuit in High Dimensions: Feasible or Infeasible Feature Extraction
- scientific article; zbMATH DE number 2042277
- Some perspectives on inference in high dimensions
- On the conditional distributions of low-dimensional projections from high-dimensional data
- Projective resampling estimation of informative predictor subspace for multivariate regression
- scientific article; zbMATH DE number 2043421
- High-dimensional variable selection with sparse random projections: measurement sparsity and statistical efficiency
Bayesian inference (62F15) Statistical ranking and selection procedures (62F07) Generalized linear models (logistic models) (62J12)
Cites Work
- Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC
- The Adaptive Lasso and Its Oracle Properties
- Least angle regression. (With discussion)
- A survey of Bayesian predictive methods for model assessment, selection and comparison
- Comparison of Bayesian predictive methods for model selection
- Bayesian data analysis.
- Title not available (Why is that?)
- Title not available (Why is that?)
- Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties
- Selection bias in gene extraction on the basis of microarray gene-expression data
- 10.1162/153244303322753715
- Title not available (Why is that?)
- The Dantzig selector: statistical estimation when \(p\) is much larger than \(n\). (With discussions and rejoinder).
- A study of error variance estimation in Lasso regression
- The horseshoe estimator for sparse signals
- Sure Independence Screening for Ultrahigh Dimensional Feature Space
- Regression modeling strategies. With applications to linear models, logistic regression, and survival analysis
- Regularization and Variable Selection Via the Elastic Net
- Prediction by Supervised Principal Components
- The horseshoe+ estimator of ultra-sparse signals
- Optimal predictive model selection.
- Needles and straw in haystacks: Empirical Bayes estimates of possibly sparse sequences
- Model choice in generalised linear models: a Bayesian approach via Kullback-Leibler projections
- Bayesian Model Averaging for Linear Regression Models
- Title not available (Why is that?)
- Dirichlet-Laplace priors for optimal shrinkage
- Sparsity information and regularization in the horseshoe and other shrinkage priors
- Relaxed Lasso
- Bayesian variable selection with shrinking and diffusing priors
- Spike and slab variable selection: frequentist and Bayesian strategies
- Better Subset Regression Using the Nonnegative Garrote
- Bayesian model selection in high-dimensional settings
- Large-scale inference. Empirical Bayes methods for estimation, testing, and prediction
- The horseshoe estimator: posterior concentration around nearly black vectors
- Needles and straw in a haystack: posterior concentration for possibly sparse sequences
- ``Preconditioning for feature selection and regression in high-dimensional problems
- Title not available (Why is that?)
- On over-fitting in model selection and subsequent selection bias in performance evaluation
- Variable selection in qualitative models via an entropic explanatory power
- Bayesian projection approaches to variable selection in generalized linear models
- The predictive Lasso
- Using stacking to average Bayesian predictive distributions (with discussion)
- Implicitly adaptive importance sampling
- Decoupling shrinkage and selection in Bayesian linear models: a posterior summary perspective
- Scalable Importance Tempering and Bayesian Variable Selection
- A decision-theoretic approach for model interpretability in Bayesian framework
Cited In (10)
- Structured Shrinkage Priors
- Bayesian estimation of subset threshold autoregressions: short-term forecasting of traffic occupancy
- Efficient estimation and correction of selection-induced bias with order statistics
- Intuitive joint priors for Bayesian linear multilevel models: the R2D2M2 prior
- A fully Bayesian sparse polynomial chaos expansion approach with joint priors on the coefficients and global selection of terms
- Fast, Optimal, and Targeted Predictions Using Parameterized Decision Analysis
- Projective resampling estimation of informative predictor subspace for multivariate regression
- Targeted random projection for prediction from high-dimensional features
- Using reference models in variable selection
- A decision-theoretic approach for model interpretability in Bayesian framework
Uses Software
This page was built for publication: Projective inference in high-dimensional problems: prediction and feature selection
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2188473)
