Classification with many classes: challenges and pluses

DOI10.1016/J.JMVA.2019.104536MaRDI QIDQ2008227zbMATH OpenOpenAlexFDO

Authors Marianna Pensky, Felix P. Abramovich

Publication date 22 November 2019

Published in Journal of Multivariate Analysis (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1506.01567

zbMATH Keywords

feature selectionsparsityhigh-dimensionalitymisclassification errormulti-class classification

Mathematics Subject Classification ID

Bayesian inference (62F15) Classification and discrimination; cluster analysis (statistical aspects) (62H30)

Abstract: The objective of the paper is to study accuracy of multi-class classification in high-dimensional setting, where the number of classes is also large ("large $L$, large $p$, small $n$" model). While this problem arises in many practical applications and many techniques have been recently developed for its solution, to the best of our knowledge nobody provided a rigorous theoretical analysis of this important setup. The purpose of the present paper is to fill in this gap. We consider one of the most common settings, classification of high-dimensional normal vectors where, unlike standard assumptions, the number of classes could be large. We derive non-asymptotic conditions on effects of significant features, and the low and the upper bounds for distances between classes required for successful feature selection and classification with a given accuracy. Furthermore, we study an asymptotic setup where the number of classes is diverging with the dimension of feature space and while the number of samples per class is possibly limited. We point out on an interesting and, at first glance, somewhat counter-intuitive phenomenon that a large number of classes may be a "blessing" rather than a "curse" since, in certain settings, the precision of classification can improve as the number of classes grows. This is due to more accurate feature selection since even weaker significant features, which are not sufficiently strong to be manifested in a coarse classification, being shared across the classes, have a stronger impact as the number of classes increases. We supplement our theoretical investigation by a simulation study and a real data example where we again observe the above phenomenon.

Recommendations

• Model selection for classification with a large number of classes
• High-dimensional classification when useful information comes from many, perhaps all features
• Impossibility of successful classification when useful features are rare and weak
• Bayesian feature selection for classification with possibly large number of classes
• A procedure of linear discrimination analysis with detected sparsity structure for high-dimensional multi-class classification

Cites work

• scientific article; zbMATH DE number 6388313 (Why is no real title available?)
• scientific article; zbMATH DE number 3733065 (Why is no real title available?)
• 10.1162/15324430260185628
• A framework for kernel-based multi-category classification
• An alternative point of view on Lepski's method
• Bayesian feature selection for classification with possibly large number of classes
• Classification of sparse high-dimensional vectors
• Computing the distribution of quadratic forms in normal variables
• Feature selection by higher criticism thresholding achieves the optimal phase diagram
• Global testing under sparse alternatives: ANOVA, multiple comparisons and the higher criticism
• High-dimensional classification using features annealed independence rules
• Impossibility of successful classification when useful features are rare and weak
• Multi-class classification in image analysis via error-correcting output codes
• Multicategory Support Vector Machines
• On the consistency of multiclass classification methods
• Some theory for Fisher's linear discriminant function, `naive Bayes', and some alternatives when there are many more variables than observations
• Sparse linear discriminant analysis by thresholding for high dimensional data
• Theory of Classification: a Survey of Some Recent Advances
• Training highly multiclass classifiers

Cited in

(5)

• Optimal discriminant analysis in high-dimensional latent factor models
• Model selection for classification with a large number of classes
• Impossibility of successful classification when useful features are rare and weak
• Extrapolating expected accuracies for large multi-class problems
• Training highly multiclass classifiers

This page was built for publication: Classification with many classes: challenges and pluses

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2008227)