Tracy-Widom at each edge of real covariance and MANOVA estimators (Q2083270)

Consider a deterministic diagonal matrix \(T=\operatorname{diag}(t_1,\dots,t_M)\) (real, of size \(M\times M\)). Let \(X\) be a random matrix of size \(M\times N\) with independent entries with mean \(0\) and variance \(1/N\). The authors study the matrix \(\hat{\Sigma} = X'TX\) in the limit when \(M,N\to\infty\) proportionally (i.e., \(M/N\) is bounded away from \(0\) and \(\infty\)). The empirical spectrum of \(\hat{\Sigma}\) can be well approximated by a deterministic distribution \(\mu_0\). For example, if \(T\) is the identity matrix, then \(\mu_0\) is the Marchenko-Pastur distribution; in general, \(\mu_0\) can be defined implicitly: its Stieltjes transform \(m_0\) satisfies a certain equation. The support of \(\mu_0\) can consist of multiple disjoint intervals, the endpoints of which are called edges (their location is encoded by some properties of \(m_0\)). Under certain assumptions, all eigenvalues of the random matrix \(\hat{\Sigma}\) are close to the support of \(\mu_0\) (with high probability, for sufficiently large \(N\)). Thus, one may consider the smallest and largest eigenvalue of \(\hat{\Sigma}\) near a given edge and study their fluctuations. The authors prove that the appropriately normalized distance of the extremal eigenvalue to the edge converges in distribution to the Tracy-Widom law (under some assumptions on the regularity of the edge). The proof builds on and extends the work of \textit{J. O. Lee} and \textit{K. Schnelli} [Ann. Appl. Probab. 26, 3786--3839 (2016; Zbl 1384.60026)], who proved a similar result for the largest eigenvalue in the case when all the eigenvalues of the diagonal matrix \(T\) are positive. The result has applications to linear mixed models and MANOVA-type covariance estimators.