Eigenvalue distributions of variance components estimators in high-dimensional random effects models (Q2328062)

Motivated by evolutionary genetics, the authors consider the multivariate mixed effects model \[ Y=X\beta+\sum_{r=1}^kU_r\alpha_r\,, \] where \(Y\in\mathbb{R}^{n\times p}\) represents \(n\) observations of \(p\) traits, which is modelled as a sum of fixed effects \(X\beta\) (with \(X\in\mathbb{R}^{n\times m}\) a design matrix and \(\beta\in\mathbb{R}^{m\times p}\) a matrix of unknown coefficients) and random effects \(U_1\alpha_1,\ldots,U_k\alpha_k\), where \(U_r\in\mathbb{R}^{n\times I_r}\) are known incidence matrices and \(\alpha_r\sim\mathcal{N}(0,\mbox{Id}_{I_r}\otimes\Sigma_r)\) are \(I_r\times p\) random matrices with i.i.d. rows having Gaussian distributions with unknown variance parameters \(\Sigma_r\in\mathbb{R}^{p\times p}\). Their interest is particularly in asymptotic (\(n\rightarrow\infty\)) results in the case where \(p\) and \(I_1,\ldots,I_k\) grow proportionally to \(n\). In this setting, the authors study estimators of the variance components in this model of the form \[ \hat{\Sigma}_r=Y^\top B_r Y\,, \] for symmetric matrices \(B_r\in\mathbb{R}^{n\times n}\) with \(B_rX=0\). This includes MANOVA estimators and MINQUEs. The main result of the paper is that the empirical spectra of these estimators converge to a deterministic limit defined by a generalization of the Marcenko--Pastur equation. The proof uses operator-valued free probability, and includes a general asymptotic freeness result for certain random matrices. The authors also give an explicit algorithm for the computation of this limit, and show how their results specialize to the setting of one-way classification, a particular case of the above general model.