Feature selection by higher criticism thresholding achieves the optimal phase diagram

DOI10.1098/rsta.2009.0129zbMath1185.62113arXiv0812.2263OpenAlexW3100205528WikidataQ51787285 ScholiaQ51787285MaRDI QIDQ3559955

David L. Donoho, Jiashun Jin

Publication date: 8 May 2010

Published in: Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences (Search for Journal in Brave)

Full work available at URL: https://arxiv.org/abs/0812.2263

zbMATH Keywords

phase diagramfalse discovery ratelinear classificationasymptotic rare/weak modelfeature selection by thresholdingFisher's separation measure

Mathematics Subject Classification ID

Classification and discrimination; cluster analysis (statistical aspects) (62H30) Order statistics; empirical distribution functions (62G30)

Related Items (26)

Goodness of fit tests in terms of local levels with special emphasis on higher criticism tests ⋮ Adaptive threshold-based classification of sparse high-dimensional data ⋮ Higher criticism to compare two large frequency tables, with sensitivity to possible rare and weak differences ⋮ High dimensional classifiers in the imbalanced case ⋮ Sparse microwave imaging: principles and applications ⋮ The intermediates take it all: Asymptotics of higher criticism statistics and a powerful alternative based on equal local levels ⋮ Feature selection when there are many influential features ⋮ Signal localization: a new approach in signal discovery ⋮ Identifying the support of rectangular signals in Gaussian noise ⋮ Estimating the amount of sparsity in two-point mixture models ⋮ Detection boundary in sparse regression ⋮ Optimal classification in sparse Gaussian graphic model ⋮ Asymptotics of goodness-of-fit tests based on minimum p-value statistics ⋮ Observed universality of phase transitions in high-dimensional geometry, with implications for modern data analysis and signal processing ⋮ Classification of sparse high-dimensional vectors ⋮ Fast rate of convergence in high-dimensional linear discriminant analysis ⋮ Classification with many classes: challenges and pluses ⋮ Innovated higher criticism for detecting sparse signals in correlated noise ⋮ Two-group classification with high-dimensional correlated data: a factor model approach ⋮ Higher Criticism for Discriminating Word-Frequency Tables and Testing Authorship ⋮ Goodness-of-Fit Tests Based on Sup-Functionals of Weighted Empirical Processes ⋮ Tight conditions for consistency of variable selection in the context of high dimensionality ⋮ Optimal Detection of Heterogeneous and Heteroscedastic Mixtures ⋮ Signal detection via Phi-divergences for general mixtures ⋮ Higher criticism for large-scale inference, especially for rare and weak effects ⋮ Using visual statistical inference to better understand random class separations in high dimension, low sample size data

Cites Work

• High-dimensional classification using features annealed independence rules
• Some theory for Fisher's linear discriminant function, `naive Bayes', and some alternatives when there are many more variables than observations
• Higher criticism for detecting sparse heterogeneous mixtures.
• Needles and straw in haystacks: Empirical Bayes estimates of possibly sparse sequences
• Asymptotic minimaxity of false discovery rate thresholding for sparse exponential data
• Goodness-of-fit tests via phi-divergences
• Estimation and confidence sets for sparse normal mixtures
• Properties of higher criticism under strong dependence
• Adapting to unknown sparsity by controlling the false discovery rate
• Higher criticism thresholding: Optimal feature selection when useful features are rare and weak
• Impossibility of successful classification when useful features are rare and weak
• Classification of sparse high-dimensional vectors

This page was built for publication: Feature selection by higher criticism thresholding achieves the optimal phase diagram