Correlation and variable importance in random forests

DOI10.48550/ARXIV.1310.5726zbMATH Open1505.62167arXiv1310.5726OpenAlexW2169281690MaRDI QIDQ58729FDOQ58729

Baptiste Gregorutti, Bertrand Michel, Philippe Saint-Pierre, Philippe Saint-Pierre, Bertrand Michel, Baptiste Gregorutti

Publication date: 21 October 2013

Published in: Statistics and Computing (Search for Journal in Brave)

Abstract: This paper is about variable selection with the random forests algorithm in presence of correlated predictors. In high-dimensional regression or classification frameworks, variable selection is a difficult task, that becomes even more challenging in the presence of highly correlated predictors. Firstly we provide a theoretical study of the permutation importance measure for an additive regression model. This allows us to describe how the correlation between predictors impacts the permutation importance. Our results motivate the use of the Recursive Feature Elimination (RFE) algorithm for variable selection in this context. This algorithm recursively eliminates the variables using permutation importance measure as a ranking criterion. Next various simulation experiments illustrate the efficiency of the RFE algorithm for selecting a small number of variables together with a good prediction error. Finally, this selection algorithm is tested on the Landsat Satellite data from the UCI Machine Learning Repository.

Full work available at URL: https://arxiv.org/abs/1310.5726

Recommendations

• A new variable selection approach using random forests
• Empirical characterization of random forest variable importance measures
• A computationally fast variable importance test for random forests for high-dimensional data
• Random forests
• Variable importance in binary regression trees and forests

zbMATH Keywords

random forestssupervised learningvariable importancevariable selection

Mathematics Subject Classification ID

Computational methods for problems pertaining to statistics (62-08) Classification and discrimination; cluster analysis (statistical aspects) (62H30) Learning and adaptive systems in artificial intelligence (68T05)

Cites Work

• Variable selection in kernel Fisher discriminant analysis by means of recursive feature elimina\-tion
• Empirical characterization of random forest variable importance measures
• Selection of relevant features and examples in machine learning
• Wrappers for feature subset selection
• A new variable selection approach using random forests
• Variable importance in binary regression trees and forests
• Consistency of random forests
• Title not available (Why is that?)
• Title not available (Why is that?)
• Correlated variables in regression: clustering and sparse estimation
• Selection bias in gene extraction on the basis of microarray gene-expression data
• Stability Selection
• 10.1162/153244303322753616
• 10.1162/153244303322753643
• 10.1162/153244303322753706
• 10.1162/153244303322753715
• Title not available (Why is that?)
• Reinforcement Learning Trees
• Linear Statistical Inference and its Applications
• Random forests
• Gene selection for cancer classification using support vector machines
• Variable selection in model-based discriminant analysis
• Bagging predictors
• Robust \(H_{\infty }\) control for nonlinear uncertain stochastic T-S fuzzy systems with time delays

Cited In (23)

• Measuring regional effects of model inputs with random Forest
• Random forest-based approach for physiological functional variable selection for driver's stress level classification
• Grouped variable importance with random forests and application to multiple functional data analysis
• Variable importance in binary regression trees and forests
• Variable selection and importance in presence of high collinearity: an application to the prediction of lean body mass from multi-frequency bioelectrical impedance
• Health care fraud classifiers in practice
• Trade-off between predictive performance and FDR control for high-dimensional Gaussian model selection
• Comments on ``Data science, big data and statistics
• Information criteria for model selection
• A random forest guided tour
• Measuring the Algorithmic Convergence of Randomized Ensembles: The Regression Setting
• Forward variable selection for random forest models
• All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously
• Ordinal trees and random forests: score-free recursive partitioning and improved ensembles
• Kernel-based measure of variable importance for genetic association studies
• moreparty
• Standard errors and confidence intervals for variable importance in random forest regression, classification, and survival
• A Study of Strength and Correlation in Random Forests
• Trees, forests, and impurity-based variable importance in regression
• Consistent and unbiased variable selection under indepedent features using random forest permutation importance
• Empirical characterization of random forest variable importance measures
• Understanding complex predictive models with ghost variables
• A computationally fast variable importance test for random forests for high-dimensional data

This page was built for publication: Correlation and variable importance in random forests

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q58729)