Is a Classification Procedure Good Enough?—A Goodness-of-Fit Assessment Tool for Classification Learning
From MaRDI portal
Abstract: In recent years, many non-traditional classification methods, such as Random Forest, Boosting, and neural network, have been widely used in applications. Their performance is typically measured in terms of classification accuracy. While the classification error rate and the like are important, they do not address a fundamental question: Is the classification method underfitted? To our best knowledge, there is no existing method that can assess the goodness-of-fit of a general classification procedure. Indeed, the lack of a parametric assumption makes it challenging to construct proper tests. To overcome this difficulty, we propose a methodology called BAGofT that splits the data into a training set and a validation set. First, the classification procedure to assess is applied to the training set, which is also used to adaptively find a data grouping that reveals the most severe regions of underfitting. Then, based on this grouping, we calculate a test statistic by comparing the estimated success probabilities and the actual observed responses from the validation set. The data splitting guarantees that the size of the test is controlled under the null hypothesis, and the power of the test goes to one as the sample size increases under the alternative hypothesis. For testing parametric classification models, the BAGofT has a broader scope than the existing methods since it is not restricted to specific parametric models (e.g., logistic regression). Extensive simulation studies show the utility of the BAGofT when assessing general classification procedures and its strengths over some existing methods when testing parametric classification models.
Cites work
- A comparison of the Hosmer-Lemeshow, Pigeon-Heyse, and Tsiatis goodness-of-fit tests for binary logistic regression under two grouping methods
- A Goodness-of-Fit Test for Binary Regression Models, Based on Smoothing Methods
- An Improved Goodness of Fit Statistic for Probability Prediction Models
- An omnibus lack of fit test in logistic regression with sparse data
- Cross-validation with confidence
- Goodness of fit tests for the multiple logistic regression model
- Goodness-of-fit Testing in High Dimensional Generalized Linear Models
- scientific article; zbMATH DE number 927325 (Why is no real title available?)
- scientific article; zbMATH DE number 5056254 (Why is no real title available?)
- Increasing the power: a practical approach to goodness-of-fit test for logistic regression models with continuous predictors
- Joint variable selection for fixed and random effects in linear mixed-effects models
- Maximum Likelihood Estimation of Misspecified Models
- Measuring and testing dependence by correlation of distances
- Minimax nonparametric classification .I. Rates of convergence
- Normal Goodness-of-Fit Tests for Multinomial Models with Large Degrees of Freedom
- On assessing binary regression models based on ungrouped data
- On the Asymptotic Distribution of Pearson's Statistic in Linear Exponential-Family Models
- Pearson-type goodness-of-fit test with bootstrap maximum likelihood estimation
- Testing goodness-of-fit in logistic case-control studies
Cited in
(2)
This page was built for publication: Is a Classification Procedure Good Enough?—A Goodness-of-Fit Assessment Tool for Classification Learning
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6165300)
