Rapid evaluation of the spectral signal detection threshold and Stieltjes transform

DOI10.1007/S10444-021-09890-7zbMATH Open1490.65122arXiv1904.11665OpenAlexW3195187607MaRDI QIDQ2230693FDOQ2230693

Authors: William Leeb 

Publication date: 28 September 2021

Published in: Advances in Computational Mathematics (Search for Journal in Brave)

Abstract: Accurate detection of signal components is a frequently-encountered challenge in statistical applications with low signal-to-noise ratio. This problem is particularly challenging in settings with heteroscedastic noise. In certain signal-plus-noise models of data, such as the classical spiked covariance model and its variants, there are closed formulas for the spectral signal detection threshold (the largest sample eigenvalue attributable solely to noise) for isotropic noise in the limit of infinitely large data matrices. However, more general noise models currently lack provably fast and accurate methods for numerically evaluating the threshold. In this work, we introduce a rapid algorithm for evaluating the spectral signal detection threshold in the limit of infinitely large data matrices. We consider noise matrices with a separable variance profile (whose variance matrix is rank one), as these arise often in applications. The solution is based on nested applications of Newton's method. We also devise a new algorithm for evaluating the Stieltjes transform of the spectral distribution at real values exceeding the threshold. The Stieltjes transform on this domain is known to be a key quantity in parameter estimation for spectral denoising methods. The correctness of both algorithms is proven from a detailed analysis of the master equations characterizing the Stieltjes transform, and their performance is demonstrated in numerical experiments.

Full work available at URL: https://arxiv.org/abs/1904.11665

Recommendations

• Detection threshold for non-parametric estimation
• Optimal signal detection in some spiked random matrix models: likelihood ratio tests and linear spectral statistics
• Detection of weak signals in high-dimensional complex-valued data
• On detection of the number of signals when the noise covariance matrix is arbitrary
• Fundamental limits of detection in the spiked Wigner model

zbMATH Keywords

Stieltjes transformlimiting spectral distributionNewton's methodseparable variance profilespectral signal detection threshold

Mathematics Subject Classification ID

Numerical optimization and variational techniques (65K10) Signal theory (characterization, reconstruction, filtering, etc.) (94A12) Random matrices (probabilistic aspects) (60B20) Newton-type methods (49M15)

Cites Work

• Efficient computation of limit spectra of sample covariance matrices
• The polynomial method for random matrices
• OptShrink: An Algorithm for Improved Low-Rank Signal Matrix Denoising by Optimal, Data-Driven Singular Value Shrinkage
• Numerical implementation of the QuEST function
• On the distribution of the largest eigenvalue in principal components analysis
• Strong convergence of the empirical distribution of eigenvalues of large dimensional random matrices
• Asymptotics of sample eigenstructure for a large dimensional spiked covariance model
• A rationale and test for the number of factors in factor analysis
• The singular values and vectors of low rank perturbations of large rectangular random matrices
• Spectrum estimation for large dimensional covariance matrices using random matrix theory
• Title not available (Why is that?)
• Title not available (Why is that?)
• Numerical Methods in Scientific Computing, Volume I
• Spectrum estimation: a unified framework for covariance matrix estimation and PCA in large dimensions
• Reconstruction of a low-rank matrix in the presence of Gaussian noise
• On the empirical distribution of eigenvalues of a class of large dimensional random matrices
• Asymptotic power of sphericity tests for high-dimensional data
• Random matrix methods for wireless communications.
• Optimal shrinkage of eigenvalues in the spiked covariance model
• Optimal Shrinkage of Singular Values
• Analysis of the limiting spectral measure of large random matrices of the separable covariance type
• No eigenvalues outside the support of the limiting empirical spectral distribution of a separable covariance matrix
• A limit theorem for the norm of random matrices
• Non-Parametric Detection of the Number of Signals: Hypothesis Testing and Random Matrix Theory
• Asymptotic performance of PCA for high-dimensional heteroscedastic data
• Lectures on convex optimization
• On determining the number of spikes in a high-dimensional spiked population model
• MIXANDMIX: numerical techniques for the computation of empirical spectral distributions of population mixtures
• Deterministic parallel analysis: an improved method for selecting factors and principal components
• Spiked separable covariance matrices and principal components
• Permutation methods for factor analysis and PCA
• Singular vector and singular subspace distribution for the matrix denoising model
• Optimal prediction in the linearly transformed spiked model
• Optimal Spectral Shrinkage and PCA With Heteroscedastic Noise
• Matrix denoising for weighted loss functions and heterogeneous signals

Cited In (3)

• Matrix denoising for weighted loss functions and heterogeneous signals
• Biwhitening Reveals the Rank of a Count Matrix
• Data-driven optimal shrinkage of singular values under high-dimensional noise with separable covariance structure with application

Uses Software

• SPECTRODE
• RMTool
• OptShrink
• QuEST

This page was built for publication: Rapid evaluation of the spectral signal detection threshold and Stieltjes transform

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2230693)