Spectrum estimation for large dimensional covariance matrices using random matrix theory

DOI10.1214/07-AOS581zbMath1168.62052arXivmath/0609418MaRDI QIDQ1000306

Noureddine El Karoui

Publication date: 6 February 2009

Published in: The Annals of Statistics (Search for Journal in Brave)

Full work available at URL: https://arxiv.org/abs/math/0609418

zbMATH Keywords

Stieltjes transformconvex optimizationhigh-dimensional inferenceprincipal components analysisMarčenko-Pastur equationeigenvalues of covariance matrices

Mathematics Subject Classification ID

Factor analysis and principal components; correspondence analysis (62H25) Estimation in multivariate analysis (62H12) Convex programming (90C25) Eigenvalues, singular values, and eigenvectors (15A18) Random matrices (algebraic aspects) (15B52) Graphical methods in statistics (62A09)

Related Items (78)

On the non-local priors for sparsity selection in high-dimensional Gaussian DAG models ⋮ An orthogonally equivariant estimator of the covariance matrix in high dimensions and for small sample sizes ⋮ Almost sure convergence of the largest and smallest eigenvalues of high-dimensional sample correlation matrices ⋮ Analysis of overfitting in the regularized Cox model ⋮ The role of the isotonizing algorithm in Stein's covariance matrix estimator ⋮ Recent developments in high dimensional covariance estimation and its related issues, a review ⋮ Numerical implementation of the QuEST function ⋮ Testing the order of a population spectral distribution for high-dimensional data ⋮ A permutation-based Bayesian approach for inverse covariance estimation ⋮ Unnamed Item ⋮ Optimal regularizations for data generation with probabilistic graphical models ⋮ Statistical significance in high-dimensional linear models ⋮ Optimal shrinkage of eigenvalues in the spiked covariance model ⋮ Reconstruction of a low-rank matrix in the presence of Gaussian noise ⋮ Estimation of the population spectral distribution from a large dimensional sample covariance matrix ⋮ High-dimensional covariance matrix estimation with missing observations ⋮ Matrix means and a novel high-dimensional shrinkage phenomenon ⋮ Convergence and prediction of principal component scores in high-dimensional settings ⋮ Consistent estimation of high-dimensional factor models when the factor number is over-estimated ⋮ Optimal cleaning for singular values of cross-covariance matrices ⋮ Efficient estimation of approximate factor models via penalized maximum likelihood ⋮ Consistent Bayesian sparsity selection for high-dimensional Gaussian DAG models with multiplicative and beta-mixture priors ⋮ Fast randomized numerical rank estimation for numerically low-rank matrices ⋮ Estimation of the inverse scatter matrix of an elliptically symmetric distribution ⋮ A bootstrap method for spectral statistics in high-dimensional elliptical models ⋮ Long random matrices and tensor unfolding ⋮ Detection of Multiple Structural Breaks in Large Covariance Matrices ⋮ Detecting the large entries of a sparse covariance matrix in sub-quadratic time ⋮ On a model selection problem from high-dimensional sample covariance matrices ⋮ Boundary behavior in high dimension, low sample size asymptotics of PCA ⋮ Inferring large graphs using \(\ell_1\)-penalized likelihood ⋮ Limit theory for the largest eigenvalues of sample covariance matrices with heavy-tails ⋮ Eigenvectors of some large sample covariance matrix ensembles ⋮ Covariance estimation: the GLM and regularization perspectives ⋮ On the estimation of integrated covariance matrices of high dimensional diffusion processes ⋮ From Synaptic Interactions to Collective Dynamics in Random Neuronal Networks Models: Critical Role of Eigenvectors and Transient Behavior ⋮ Random matrix theory in statistics: a review ⋮ Posterior graph selection and estimation consistency for high-dimensional Bayesian DAG models ⋮ Rapid evaluation of the spectral signal detection threshold and Stieltjes transform ⋮ Shrinkage estimation of large covariance matrices: keep it simple, statistician? ⋮ Joint Bayesian Variable and DAG Selection Consistency for High-dimensional Regression Models with Network-structured Covariates ⋮ An efficient numerical method for condition number constrained covariance matrix approximation ⋮ Accuracy of regularized D-rule for binary classification ⋮ Optimal estimation of a large-dimensional covariance matrix under Stein's loss ⋮ On the inference about the spectral distribution of high-dimensional covariance matrix based on high-frequency noisy observations ⋮ Cleaning large correlation matrices: tools from random matrix theory ⋮ A unified model for regularized and robust portfolio optimization ⋮ Resolvent of large random graphs ⋮ Statistical and Knowledge Supported Visualization of Multivariate Data ⋮ Subordination methods for free deconvolution ⋮ On the dimension effect of regularized linear discriminant analysis ⋮ Universal distribution of batch completion times and time-cost tradeoff in a production line with arbitrary buffer size ⋮ Asymptotic properties of principal component analysis and shrinkage-bias adjustment under the generalized spiked population model ⋮ Ridge regression and asymptotic minimax estimation over spheres of growing dimension ⋮ Covariance regularization by thresholding ⋮ Operator norm consistent estimation of large-dimensional sparse covariance matrices ⋮ Scalable estimation strategies based on stochastic approximations: classical results and new insights ⋮ Estimation of two high-dimensional covariance matrices and the spectrum of their ratio ⋮ Spiked separable covariance matrices and principal components ⋮ Nonlinear shrinkage estimation of large-dimensional covariance matrices ⋮ Local expectations of the population spectral distribution of a high-dimensional covariance matrix ⋮ Compressed covariance estimation with automated dimension learning ⋮ High-dimensional linear models: a random matrix perspective ⋮ Singular value distribution of the propagation matrix in random scattering media ⋮ Random matrix theory and its applications ⋮ Edge statistics of large dimensional deformed rectangular matrices ⋮ Eigenvalue distributions of variance components estimators in high-dimensional random effects models ⋮ Concentration of measure and spectra of random matrices: applications to correlation matrices, elliptical distributions and beyond ⋮ Correlation structure selection for longitudinal data with diverging cluster size ⋮ Random matrix improved covariance estimation for a large class of metrics* ⋮ Spiked sample covariance matrices with possibly multiple bulk components ⋮ Independence test for high dimensional data based on regularized canonical correlation coefficients ⋮ Estimation of high-dimensional dynamic conditional precision matrices with an application to forecast combination ⋮ Multiple Anchor Point Shrinkage for the Sample Covariance Matrix ⋮ Spectrum estimation: a unified framework for covariance matrix estimation and PCA in large dimensions ⋮ Replica analysis of overfitting in generalized linear regression models ⋮ On generalized expectation-based estimation of a population spectral distribution from high-dimensional data ⋮ Penalization-induced shrinking without rotation in high dimensional GLM regression: a cavity analysis

Uses Software

• Mosek
• PDCO

Cites Work

• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• A well-conditioned estimator for large-dimensional covariance matrices
• Statistical eigen-inference from large Wishart matrices
• On the limit of the largest eigenvalue of the large dimensional sample covariance matrix
• A limit theorem for the norm of random matrices
• Some limit theorems for the eigenvalues of a sample covariance matrix
• The strong limits of random matrix spectra for sample matrices of independent elements
• Introduction to large truncated Toeplitz matrices
• Some theory for Fisher's linear discriminant function, `naive Bayes', and some alternatives when there are many more variables than observations
• Limiting spectral distribution for a class of random matrices
• On the distribution of the largest eigenvalue in principal components analysis
• Necessary and sufficient condition that the limit of Stieltjes transforms is a Stieltjes transform
• On the empirical distribution of eigenvalues of a class of large dimensional random matrices
• Strong convergence of the empirical distribution of eigenvalues of large dimensional random matrices
• On the empirical distribution of eigenvalues of large dimensional information-plus-noise-type matrices
• Tracy-Widom limit for the largest eigenvalue of a large class of complex sample covariance matrices
• Regularized estimation of large covariance matrices
• Eigenvalues of large sample covariance matrices of spiked population models
• Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices
• Atomic Decomposition by Basis Pursuit
• On Choosing and Bounding Probability Metrics
• Asymptotic Theory for Principal Component Analysis
• The elements of statistical learning. Data mining, inference, and prediction

This page was built for publication: Spectrum estimation for large dimensional covariance matrices using random matrix theory