Permutation methods for factor analysis and PCA

DOI10.1214/19-AOS1907MaRDI QIDQ2215761zbMATH OpenOpenAlexFDO

Authors Edgar Dobriban

Publication date 14 December 2020

Published in The Annals of Statistics (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1710.00479, https://projecteuclid.org/euclid.aos/1600480933

zbMATH Keywords

factor analysishigh-dimensional asymptoticsparallel analysispermutation methodsprincipal component analysis (PCA)

Mathematics Subject Classification ID

Signal detection and filtering (aspects of stochastic processes) (60G35) Estimation in multivariate analysis (62H12) Factor analysis and principal components; correspondence analysis (62H25) Signal theory (characterization, reconstruction, filtering, etc.) (94A12)

Abstract: Researchers often have datasets measuring features $x_{ij}$ of samples, such as test scores of students. In factor analysis and PCA, these features are thought to be influenced by unobserved factors, such as skills. Can we determine how many components affect the data? This is an important problem, because it has a large impact on all downstream data analysis. Consequently, many approaches have been developed to address it. Parallel Analysis is a popular permutation method. It works by randomly scrambling each feature of the data. It selects components if their singular values are larger than those of the permuted data. Despite widespread use in leading textbooks and scientific publications, as well as empirical evidence for its accuracy, it currently has no theoretical justification. In this paper, we show that the parallel analysis permutation method consistently selects the large components in certain high-dimensional factor models. However, it does not select the smaller components. The intuition is that permutations keep the noise invariant, while "destroying" the low-rank signal. This provides justification for permutation methods in PCA and factor models under some conditions. Our work uncovers drawbacks of permutation methods, and paves the way to improvements.

Review text

From the author's abstract: ``Researchers often have datasets measuring features \(x_{ij}\) of samples, such as test scores of students. In factor analysis and PCA, these features are thought to be influenced by unobserved factors, such as skills. Can we determine how many components affect the data? This is an important problem, because decisions made here have a large impact on all downstream data analysis. Consequently, many approaches have been developed. Parallel Analysis is a popular permutation method: it randomly scrambles each feature of the data. It selects components if their singular values are larger than those of the permuted data. Despite widespread use, as well as empirical evidence for its accuracy, it currently has no theoretical justification. In this paper, the problem is analyzed under the signal plus noise model. Sufficient conditions on the signal components and on the noise component are established to ensure the consistency of the parallel analysis. A simulation study supports the theoretical results and points out interesting features of the parallel analysis. In particular, the effect of signal strength, the effect of delocalization, and the effect of dimension are studied. Finally, it is shown that strong signals may lead to errors in the detection of weak signal components.

Recommendations

• A new approach for selecting the number of factors
• Deterministic parallel analysis: an improved method for selecting factors and principal components
• Statistical significance of the contribution of variables to the PCA solution: an alternative permutation strategy
• On the number of principal components: a test of dimensionality based on measurements of similarity between matrices
• Stability approach to selecting the number of principal components

Cites work

• A general framework for multiple testing dependence
• A rationale and test for the number of factors in factor analysis
• Asymptotic power of sphericity tests for high-dimensional data
• Asymptotics of sample eigenstructure for a large dimensional spiked covariance model
• Asymptotics of the principal components estimator of large factor models with weakly influential factors
• Considering Horn's parallel analysis from a random matrix theory point of view
• Determining the number of components from the matrix of partial correlations
• Deterministic parallel analysis: an improved method for selecting factors and principal components
• Eigenvalue significance testing for genetic association
• Estimation of spiked eigenvalues in spiked models
• Finite sample approximation results for principal component analysis: A matrix perturbation approach
• High-dimensional asymptotics of prediction: ridge regression and classification
• How many principal components? Stopping rules for determining the number of non-trivial axes revisited
• scientific article; zbMATH DE number 3136275 (Why is no real title available?)
• scientific article; zbMATH DE number 3045859 (Why is no real title available?)
• On the distribution of the largest eigenvalue in principal components analysis
• Optimal prediction in the linearly transformed spiked model
• OptShrink: An Algorithm for Improved Low-Rank Signal Matrix Denoising by Optimal, Data-Driven Singular Value Shrinkage
• Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices
• Principal component analysis.
• Random matrix theory in statistics: a review
• Simultaneous dimension reduction and adjustment for confounding variation
• Spectral analysis of large dimensional random matrices
• Testing hypotheses about the number of factors in large factor models
• The singular values and vectors of low rank perturbations of large rectangular random matrices

Cited in

(19)

• Estimating change-point latent factor models for high-dimensional time series
• A note on the likelihood ratio test in high-dimensional exploratory factor analysis
• Consistency of invariance-based randomization tests
• Statistical inference for principal components of spiked covariance matrices
• Rapid evaluation of the spectral signal detection threshold and Stieltjes transform
• Statistical significance of the contribution of variables to the PCA solution: an alternative permutation strategy
• Deterministic parallel analysis: an improved method for selecting factors and principal components
• Permutation Statistical Methods with R
• Matrix denoising for weighted loss functions and heterogeneous signals
• Considering Horn's parallel analysis from a random matrix theory point of view
• Estimating Number of Factors by Adjusted Eigenvalues Thresholding
• Biwhitening Reveals the Rank of a Count Matrix
• A CLT for the LSS of large-dimensional sample covariance matrices with diverging spikes
• Sampling without replacement from a high-dimensional finite population
• The limiting spectral distribution of large random permutation matrices
• An ensemble approach to determine the number of latent dimensions and assess its reliability
• The Dyson equalizer: adaptive noise stabilization for low-rank signal detection and recovery
• Estimating the number of significant components in high-dimensional principal component analysis
• Inference in unbalanced panel data models with interactive fixed effects

Describes a project that uses

Uses Software

• nFactors
• ElemStatLearn
• OptShrink

This page was built for publication: Permutation methods for factor analysis and PCA

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2215761)