Testing independence with high-dimensional correlated samples (Q1750290)

A data matrix \(X=(X_1,\dots,X_n)\) is considered where each column \(X_i\in \mathbb R^p\) follows the same normal population distribution with unknown mean and covariance matrix \(\Sigma\). The problem of testing independence among the columns is studied in the situation where \(p\) is much larger than \(n\). A computationally simple and tuning-free test statistic is presented, its limiting null distribution is founded, the statistical power is analyzed, and its minimax optimality is shown. A ratio-consistent estimator for the quadratic functional of \(\Sigma\) from correlated samples is developed. The effect of correlation among samples to large-scale multiple testing of Pearson's correlation coefficients is studied. Ignoring possible independence of samples leads to false statistical inference. Instead, a sandwich estimator of Pearson's correlation coefficient is proposed by de-correlating the samples. The resulting testing procedure asymptotically controls the overall discovery rate at the nominal level with good statistical power. Simulation study and real data experiments illustrate the proposed methods.