Regularized target encoding outperforms traditional methods in supervised machine learning with high cardinality features
From MaRDI portal
Publication:2095774
Abstract: Since most machine learning (ML) algorithms are designed for numerical inputs, efficiently encoding categorical variables is a crucial aspect in data analysis. A common problem are high cardinality features, i.e. unordered categorical predictor variables with a high number of levels. We study techniques that yield numeric representations of categorical variables which can then be used in subsequent ML applications. We focus on the impact of these techniques on a subsequent algorithm's predictive performance, and -- if possible -- derive best practices on when to use which technique. We conducted a large-scale benchmark experiment, where we compared different encoding strategies together with five ML algorithms (lasso, random forest, gradient boosting, k-nearest neighbors, support vector machine) using datasets from regression, binary- and multiclass- classification settings. In our study, regularized versions of target encoding (i.e. using target predictions based on the feature levels in the training set as a new numerical feature) consistently provided the best results. Traditionally widely used encodings that make unreasonable assumptions to map levels to integers (e.g. integer encoding) or to reduce the number of levels (possibly based on target information, e.g. leaf encoding) before creating binary indicator variables (one-hot or dummy encoding) were not as effective in comparison.
Recommendations
- Model-aware categorical data embedding: a data-driven approach
- Binarsity: a penalization for one-hot encoded features in linear supervised learning
- An efficient random forests algorithm for high dimensional data classification
- The use of autoencoders for training neural networks with mixed categorical and numerical features
- Solution methods for classification problems with categorical attributes
Cites work
- Additive structure in qualitative data: An alternating least squares method with optimal scaling features
- Benchmark for filter methods for feature selection in high-dimensional classification data
- Conditional likelihood maximisation: a unifying framework for information theoretic feature selection
- Do we need hundreds of classifiers to solve real world classification problems?
- scientific article; zbMATH DE number 2033166 (Why is no real title available?)
- Inference for the generalization error
- mlr: machine learning in \(\mathbf R\)
- On the necessity and design of studies comparing statistical methods
- Regression with qualitative and quantitative variables: An alternating least squares method with optimal scaling features
Cited in
(3)
Describes a project that uses
Uses Software
This page was built for publication: Regularized target encoding outperforms traditional methods in supervised machine learning with high cardinality features
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2095774)
