Rapid evaluation of the spectral signal detection threshold and Stieltjes transform

DOI10.1007/S10444-021-09890-7zbMATH Open1490.65122OpenAlexW3195187607MaRDI QIDQ2230693FDOQ2230693

Authors William Leeb 

Publication date 28 September 2021

Published in Advances in Computational Mathematics (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1904.11665

zbMATH Keywords

Stieltjes transformlimiting spectral distributionNewton's methodseparable variance profilespectral signal detection threshold

Mathematics Subject Classification ID

Numerical optimization and variational techniques (65K10) Signal theory (characterization, reconstruction, filtering, etc.) (94A12) Random matrices (probabilistic aspects) (60B20) Newton-type methods (49M15)

Abstract: Accurate detection of signal components is a frequently-encountered challenge in statistical applications with low signal-to-noise ratio. This problem is particularly challenging in settings with heteroscedastic noise. In certain signal-plus-noise models of data, such as the classical spiked covariance model and its variants, there are closed formulas for the spectral signal detection threshold (the largest sample eigenvalue attributable solely to noise) for isotropic noise in the limit of infinitely large data matrices. However, more general noise models currently lack provably fast and accurate methods for numerically evaluating the threshold. In this work, we introduce a rapid algorithm for evaluating the spectral signal detection threshold in the limit of infinitely large data matrices. We consider noise matrices with a separable variance profile (whose variance matrix is rank one), as these arise often in applications. The solution is based on nested applications of Newton's method. We also devise a new algorithm for evaluating the Stieltjes transform of the spectral distribution at real values exceeding the threshold. The Stieltjes transform on this domain is known to be a key quantity in parameter estimation for spectral denoising methods. The correctness of both algorithms is proven from a detailed analysis of the master equations characterizing the Stieltjes transform, and their performance is demonstrated in numerical experiments.

Recommendations

• Detection threshold for non-parametric estimation
• Optimal signal detection in some spiked random matrix models: likelihood ratio tests and linear spectral statistics
• Detection of weak signals in high-dimensional complex-valued data
• On detection of the number of signals when the noise covariance matrix is arbitrary
• Fundamental limits of detection in the spiked Wigner model

Cites Work

• scientific article; zbMATH DE number 6026126 (Why is no real title available?)
• scientific article; zbMATH DE number 3244317 (Why is no real title available?)
• A limit theorem for the norm of random matrices
• A rationale and test for the number of factors in factor analysis
• Analysis of the limiting spectral measure of large random matrices of the separable covariance type
• Asymptotic performance of PCA for high-dimensional heteroscedastic data
• Asymptotic power of sphericity tests for high-dimensional data
• Asymptotics of sample eigenstructure for a large dimensional spiked covariance model
• Deterministic parallel analysis: an improved method for selecting factors and principal components
• Efficient computation of limit spectra of sample covariance matrices
• Lectures on convex optimization
• MIXANDMIX: numerical techniques for the computation of empirical spectral distributions of population mixtures
• Matrix denoising for weighted loss functions and heterogeneous signals
• No eigenvalues outside the support of the limiting empirical spectral distribution of a separable covariance matrix
• Non-Parametric Detection of the Number of Signals: Hypothesis Testing and Random Matrix Theory
• Numerical Methods in Scientific Computing, Volume I
• Numerical implementation of the QuEST function
• On determining the number of spikes in a high-dimensional spiked population model
• On the distribution of the largest eigenvalue in principal components analysis
• On the empirical distribution of eigenvalues of a class of large dimensional random matrices
• OptShrink: An Algorithm for Improved Low-Rank Signal Matrix Denoising by Optimal, Data-Driven Singular Value Shrinkage
• Optimal Shrinkage of Singular Values
• Optimal Spectral Shrinkage and PCA With Heteroscedastic Noise
• Optimal prediction in the linearly transformed spiked model
• Optimal shrinkage of eigenvalues in the spiked covariance model
• Permutation methods for factor analysis and PCA
• Random matrix methods for wireless communications.
• Reconstruction of a low-rank matrix in the presence of Gaussian noise
• Singular vector and singular subspace distribution for the matrix denoising model
• Spectrum estimation for large dimensional covariance matrices using random matrix theory
• Spectrum estimation: a unified framework for covariance matrix estimation and PCA in large dimensions
• Spiked separable covariance matrices and principal components
• Strong convergence of the empirical distribution of eigenvalues of large dimensional random matrices
• The polynomial method for random matrices
• The singular values and vectors of low rank perturbations of large rectangular random matrices

Cited In (3)

• Matrix denoising for weighted loss functions and heterogeneous signals
• Biwhitening Reveals the Rank of a Count Matrix
• Data-driven optimal shrinkage of singular values under high-dimensional noise with separable covariance structure with application

Uses Software

• SPECTRODE
• RMTool
• OptShrink
• QuEST

This page was built for publication: Rapid evaluation of the spectral signal detection threshold and Stieltjes transform

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2230693)