Forward-backward selection with early dropping
From MaRDI portal
Publication:4633015
Abstract: Forward-backward selection is one of the most basic and commonly-used feature selection algorithms available. It is also general and conceptually applicable to many different types of data. In this paper, we propose a heuristic that significantly improves its running time, while preserving predictive accuracy. The idea is to temporarily discard the variables that are conditionally independent with the outcome given the selected variable set. Depending on how those variables are reconsidered and reintroduced, this heuristic gives rise to a family of algorithms with increasingly stronger theoretical guarantees. In distributions that can be faithfully represented by Bayesian networks or maximal ancestral graphs, members of this algorithmic family are able to correctly identify the Markov blanket in the sample limit. In experiments we show that the proposed heuristic increases computational efficiency by about two orders of magnitude in high-dimensional problems, while selecting fewer variables and retaining predictive performance. Furthermore, we show that the proposed algorithm and feature selection with LASSO perform similarly when restricted to select the same number of variables, making the proposed algorithm an attractive alternative for problems where no (efficient) algorithm for LASSO exists.
Recommendations
Cites work
- scientific article; zbMATH DE number 1817585 (Why is no real title available?)
- scientific article; zbMATH DE number 48812 (Why is no real title available?)
- scientific article; zbMATH DE number 1493045 (Why is no real title available?)
- scientific article; zbMATH DE number 3444596 (Why is no real title available?)
- scientific article; zbMATH DE number 845714 (Why is no real title available?)
- scientific article; zbMATH DE number 884923 (Why is no real title available?)
- 10.1162/153244303322753616
- A greedy feature selection algorithm for big data of high dimensionality
- A stepwise regression algorithm for high-dimensional variable selection
- Adaptive Lasso and group-Lasso for functional Poisson regression
- Adjusting Stepwisep-Values in Generalized Linear Models
- Ancestral graph Markov models.
- Applied Linear Regression
- Bayesian backfitting. (With comments and a rejoinder).
- Bootstrapping the out-of-sample predictions for efficient and accurate cross-validation
- Causation, prediction, and search
- Compressed Sensing and Source Separation
- Conditional likelihood maximisation: a unifying framework for information theoretic feature selection
- Consistent model selection criteria on high dimensions
- Estimating high-dimensional directed acyclic graphs with the PC-algorithm
- Estimating the dimension of a model
- Estimation for high-dimensional linear mixed-effects models using \(\ell_1\)-penalization
- Extended Bayesian information criteria for model selection with large model spaces
- Least angle regression. (With discussion)
- Likelihood Ratio Tests for Model Selection and Non-Nested Hypotheses
- Local causal and Markov blanket induction for causal discovery and feature selection for classification. Part I: Algorithms and empirical evaluation
- Markov Properties for Acyclic Directed Mixed Graphs
- Maximum Likelihood Estimation of Misspecified Models
- Model selection principles in misspecified models
- Model selection procedure for high‐dimensional data
- Orthogonal least squares methods and their application to non-linear system identification
- Random forests
- Regression modeling strategies. With applications to linear models, logistic regression and survival analysis
- Sequential selection procedures and false discovery rate control
- Support-vector networks
- The Group Lasso for Logistic Regression
- The performance of the likelihood ratio test when the model is incorrect
- Tuning parameter selection in high dimensional penalized likelihood
- Variable selection for generalized linear mixed models by \(L_1\)-penalized estimation
Cited in
(9)- dcorVS
- Extending greedy feature selection algorithms to multiple solutions
- Features Selection as a Nash-Bargaining Solution: Applications in Online Advertising and Information Systems
- Streamwise feature selection
- A greedy feature selection algorithm for big data of high dimensionality
- Latent theme dictionary model for finding co-occurrent patterns in process data
- scientific article; zbMATH DE number 7370579 (Why is no real title available?)
- Squared error-based shrinkage estimators of discrete probabilities and their application to variable selection
- Domain knowledge-enhanced variable selection for biomedical data analysis
This page was built for publication: Forward-backward selection with early dropping
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q4633015)
