Innovated higher criticism for detecting sparse signals in correlated noise

DOI10.1214/09-AOS764zbMATH Open1189.62080arXiv0902.3837OpenAlexW3099354396MaRDI QIDQ90256FDOQ90256

Peter Hall, Jiashun Jin, Peter Hall, Jiashun Jin

Publication date: 1 June 2010

Published in: The Annals of Statistics (Search for Journal in Brave)

Abstract: Higher criticism is a method for detecting signals that are both sparse and weak. Although first proposed in cases where the noise variables are independent, higher criticism also has reasonable performance in settings where those variables are correlated. In this paper we show that, by exploiting the nature of the correlation, performance can be improved by using a modified approach which exploits the potential advantages that correlation has to offer. Indeed, it turns out that the case of independent noise is the most difficult of all, from a statistical viewpoint, and that more accurate signal detection (for a given level of signal sparsity and strength) can be obtained when correlation is present. We characterize the advantages of correlation by showing how to incorporate them into the definition of an optimal detection boundary. The boundary has particularly attractive properties when correlation decays at a polynomial rate or the correlation matrix is Toeplitz.

Full work available at URL: https://arxiv.org/abs/0902.3837

Recommendations

• Properties of higher criticism under strong dependence
• Higher criticism for detecting sparse heterogeneous mixtures.
• Detection of correlations
• Detecting a target in very noisy data from mutliple looks
• Optimal detection of multi-sample aligned sparse signals

zbMATH Keywords

adding noiseCholesky factorizationempirical processinnovationmultiple hypothesis testingsparse normal meansspectral densityToeplitz matrix

Mathematics Subject Classification ID

Statistical aspects of information-theoretic topics (62B10) Nonparametric hypothesis testing (62G10) Inference from stochastic processes (62M99) Hypothesis testing in multivariate analysis (62H15) Measures of association (correlation, canonical correlation, etc.) (62H20) Signal theory (characterization, reconstruction, filtering, etc.) (94A12)

Cites Work

• Time series: theory and methods.
• Elements of Information Theory
• Title not available (Why is that?)
• Robustness of multiple testing procedures against dependence
• A new multiple testing method in the dependent case
• Weighted empirical and quantile processes
• Introduction to large truncated Toeplitz matrices
• Some problems of hypothesis testing leading to infinitely divisible distributions
• Estimating the proportion of false null hypotheses among a large number of independently tested hypotheses
• Wiener's lemma for infinite matrices with polynomial off-diagonal decay
• Asymptotic comparison of step-down and step-up multiple test procedures based on exchangeable test statistics
• The control of the false discovery rate in multiple testing under dependency.
• Minimax detection of a signal for \(l^ n\)-balls.
• Higher criticism for detecting sparse heterogeneous mixtures.
• A stochastic process approach to false discovery control.
• Asymptotic minimaxity of false discovery rate thresholding for sparse exponential data
• Goodness-of-fit tests via phi-divergences
• Estimation and confidence sets for sparse normal mixtures
• Regularized estimation of large covariance matrices
• On false discovery control under dependence
• Properties of higher criticism under strong dependence
• Adapting to unknown sparsity by controlling the false discovery rate
• Cosmological non-Gaussian signature detection: comparing performance of different statistical tests
• Functional limit laws of Strassen and Wichura type for multiple generations of branching processes
• Higher criticism thresholding: Optimal feature selection when useful features are rare and weak
• Impossibility of successful classification when useful features are rare and weak
• Simultaneous Testing of Grouped Hypotheses: Finding Needles in Multiple Haystacks
• Title not available (Why is that?)
• Symmetry and inverse-closedness of matrix algebras and functional calculus for infinite matrices
• Testing Against a High Dimensional Alternative
• Feature selection by higher criticism thresholding achieves the optimal phase diagram
• Theoretical Measures of Relative Performance of Classifiers for High Dimensional Data with Small Sample Sizes
• Proportion of Non-Zero Normal Means: Universal Oracle Equivalences and Uniformly Consistent Estimators
• The Simes Method for Multiple Hypothesis Testing With Positively Dependent Test Statistics
• Title not available (Why is that?)
• Step-Up Multiple Testing of Parameters with Unequally Correlated Estimates
• Estimating the Null and the Proportion of Nonnull Effects in Large-Scale Multiple Comparisons
• Correlation and Large-Scale Simultaneous Significance Testing
• Title not available (Why is that?)
• Title not available (Why is that?)
• Propriétés des matrices ``bien localisées près de leur diagonale et quelques applications. (Properties of matrices ``well localized near the diagonal and some applications)

Cited In (85)

• Maximum-type tests for high-dimensional regression coefficients using Wilcoxon scores
• Goodness of fit tests in terms of local levels with special emphasis on higher criticism tests
• Detection boundary in sparse regression
• Detection boundary and higher criticism approach for rare and weak genetic effects
• Beyond HC: more sensitive tests for rare/weak alternatives
• UPS delivers optimal phase diagram in high-dimensional variable selection
• An adaptive decorrelation procedure for signal detection
• Sharp fixed \(n\) bounds and asymptotic expansions for the mean and the median of a Gaussian sample maximum, and applications to the Donoho-Jin model
• High-dimensional sparse MANOVA
• Cauchy Combination Test: A Powerful Test With Analytic p-Value Calculation Under Arbitrary Dependency Structures
• Powerful test based on conditional effects for genome-wide screening
• Detection of an anomalous cluster in a network
• Halfspace depths for scatter, concentration and shape matrices
• Optimal detection of weak positive latent dependence between two sequences of multiple tests
• The general goodness-of-fit tests for correlated data
• A Cramér moderate deviation theorem for Hotelling's \(T^{2}\)-statistic with applications to global tests
• Model selection in regression under structural constraints
• Testing degree corrections in stochastic block models
• Hypothesis testing for high-dimensional sparse binary regression
• Asymptotically independent U-statistics in high-dimensional testing
• Detecting positive correlations in a multivariate sample
• Global testing against sparse alternatives in time-frequency analysis
• Two-sample test for sparse high-dimensional multinomial distributions
• Community detection in dense random networks
• Contraction and optimality properties of an adaptive Legendre-Galerkin method: the multi-dimensional case
• Detecting rare and faint signals via thresholding maximum likelihood estimators
• Fundamental limits of exact support recovery in high dimensions
• Large-scale inference with block structure
• Detection thresholds for the \(\beta\)-model on sparse graphs
• On the statistical properties of SGoF multitesting method
• Optimal Detection of Heterogeneous and Heteroscedastic Mixtures
• Distribution-free tests for sparse heterogeneous mixtures
• Blessing of massive scale: spatial graphical model estimation with a total cardinality constraint approach
• Minimax rates in sparse, high-dimensional change point detection
• Only closed testing procedures are admissible for controlling false discovery proportions
• Optimal classification in sparse Gaussian graphic model
• Tests alternative to higher criticism for high-dimensional means under sparsity and column-wise dependence
• Sign-based test for mean vector in high-dimensional and sparse settings
• Distribution-Free Detection of Structured Anomalies: Permutation and Rank-Based Scans
• Global testing against sparse alternatives under Ising models
• Variable selection with Hamming loss
• Innovated interaction screening for high-dimensional nonlinear classification
• MAP model selection in Gaussian regression
• Some high-dimensional one-sample tests based on functions of interpoint distances
• Localized nonlinear functional equations and two sampling problems in signal processing
• planningML
• Comment
• Global testing under sparse alternatives: ANOVA, multiple comparisons and the higher criticism
• Asymptotic optimality of the Westfall-Young permutation procedure for multiple testing under dependence
• The intermediates take it all: Asymptotics of higher criticism statistics and a powerful alternative based on equal local levels
• Testing regression coefficients in high-dimensional and sparse settings
• False Discovery Rate Control Under General Dependence By Symmetrized Data Aggregation
• Higher Criticism for Discriminating Word-Frequency Tables and Testing Authorship
• Higher criticism for large-scale inference, especially for rare and weak effects
• Detection of sparse positive dependence
• Localization of matrix factorizations
• Optimal detection of multi-sample aligned sparse signals
• Sharp optimality for high-dimensional covariance testing under sparse signals
• Bump detection in the presence of dependency: does it ease or does it load?
• Detection of correlations
• Statistical limits of sparse mixture detection
• Joint Mean and Covariance Estimation with Unreplicated Matrix-Variate Data
• A tuning-free efficient test for marginal linear effects in high-dimensional quantile regression
• Sparse signal detection in heteroscedastic Gaussian sequence models: sharp minimax rates
• Fast calculation of p-values for one-sided Kolmogorov-Smirnov type statistics
• Power enhancement for testing multi-factor asset pricing models via Fisher's method
• Anomaly Detection for a Large Number of Streams: A Permutation-Based Higher Criticism Approach
• Structured Correlation Detection with Application to Colocalization Analysis in Dual-Channel Fluorescence Microscopic Imaging
• Model Selection for Classification with a Large Number of Classes
• Signal-noise ratio of genetic associations and statistical power of SNP-set tests
• Cellwise outlier detection with false discovery rate control
• Sharp multiple testing boundary for sparse sequences
• The impossibility region for detecting sparse mixtures using the higher criticism
• A combined SR-CUSUM procedure for detecting common changes in panel data
• Moderate-Dimensional Inferences on Quadratic Functionals in Ordinary Least Squares
• Sparse equisigned PCA: algorithms and performance bounds in the noisy rank-1 setting
• Identifying the support of rectangular signals in Gaussian noise
• Testing and signal identification for two-sample high-dimensional covariances via multi-level thresholding
• Validating approximate slope homogeneity in large panels
• Mean tests for high-dimensional time series
• Fast rate of convergence in high-dimensional linear discriminant analysis
• Diagonally Dominant Principal Component Analysis
• Most powerful test against a sequence of high dimensional local alternatives
• Statistical proof? The problem of irreproducibility
• Detecting structured signals in Ising models

This page was built for publication: Innovated higher criticism for detecting sparse signals in correlated noise

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q90256)