Distributed estimation of principal eigenspaces
From MaRDI portal
Publication:2284361
Inequalities; stochastic orderings (60E15) Characterization and structure theory of statistical distributions (62E10) Statistics of extreme values; tail inference (62G32) Factor analysis and principal components; correspondence analysis (62H25) Statistical aspects of big data and data science (62R07) Learning and adaptive systems in artificial intelligence (68T05)
Abstract: Principal component analysis (PCA) is fundamental to statistical machine learning. It extracts latent principal factors that contribute to the most variation of the data. When data are stored across multiple machines, however, communication cost can prohibit the computation of PCA in a central location and distributed algorithms for PCA are thus needed. This paper proposes and studies a distributed PCA algorithm: each node machine computes the top eigenvectors and transmits them to the central server; the central server then aggregates the information from all the node machines and conducts a PCA based on the aggregated information. We investigate the bias and variance for the resulting distributed estimator of the top eigenvectors. In particular, we show that for distributions with symmetric innovation, the empirical top eigenspaces are unbiased and hence the distributed PCA is "unbiased". We derive the rate of convergence for distributed PCA estimators, which depends explicitly on the effective rank of covariance, eigen-gap, and the number of machines. We show that when the number of machines is not unreasonably large, the distributed PCA performs as well as the whole sample PCA, even without full access of whole data. The theoretical results are verified by an extensive simulation study. We also extend our analysis to the heterogeneous case where the population covariance matrices are different across local machines but share similar top eigen-structures.
The authors deal with the problem of large data sets being scattered across distant places. Massive datasets are nowadays ubiquitous and interesting examples are motivating the authors, as the data recorded by IT companies from all around the world (which cannot be stored in a single data center) or health records that are scattered across many hospitals or countries. The fusion or aggregation of such data sets is extremely difficult due to communication cost, privacy, data security, ownerships and other factors. A typical approach is based on distributed statistical/regression methods that first calculate local statistics based on each subdataset and then combine all the subsample-based statistics to produce an aggregated statistic. Principal component analysis (PCA) as a tool in statistical machine learning deals in the recent literature with a certain sparsity on top eigenvectors imposed to overcome the noise accumulation. Distributed PCA needs to handle data that are partitioned and stored across multiple servers. The paper contains a distributed algorithm for estimating the top eigenvectors, the statistical error rates of the aggregated estimator and simulation results to validate the theories, under sub-Gaussian assumptions of the data (where the tails are dominated by the tails of a Gaussian). Further interesting research directions are mentioned, for instance to investigate the possibility to use heavy-tailed distributions (heavier tails than sub-Gaussian tails), when to establish statistical rate with exponential deviation require shrinkage of the data and the control of the induced bias.
Recommendations
- Distributed Estimation for Principal Component Analysis: An Enlarged Eigenspace Analysis
- Communication-Efficient Distributed Eigenspace Estimation
- Distributed estimation of Laplacian eigenvalues via constrained consensus optimization problems
- Robust covariance estimation for distributed principal component analysis
- Efficient distributed estimation of high-dimensional sparse precision matrix
- Distributed Estimation of Gaussian Correlations
- Distributed estimation and quantization
- Optimal/Near-Optimal Dimensionality Reduction for Distributed Estimation in Homogeneous and Certain Inhomogeneous Scenarios
- Decentralized estimation of Laplacian eigenvalues in multi-agent systems
- Distributed Least Mean-Square Estimation With Partial Diffusion
Cites work
- A Distributed Framework for Dimensionality Reduction and Denoising
- A Second-Order Perturbation Expansion for the SVD
- A split-and-conquer approach for analysis of
- A useful variant of the Davis-Kahan theorem for statisticians
- Asymptotic Theory for Principal Component Analysis
- Asymptotics and concentration bounds for bilinear forms of spectral projectors of sample covariance
- Asymptotics of empirical eigenstructure for high dimensional spiked covariance
- Asymptotics of sample eigenstructure for a large dimensional spiked covariance model
- Asymptotics of the principal components estimator of large factor models with weakly influential factors
- Communication-efficient sparse regression
- Concentration inequalities and moment bounds for sample covariance operators
- Consistency of sparse PCA in high dimension, low sample size contexts
- Distributed clustering using collective principal component analysis
- Distributed estimation of principal eigenspaces
- Finding structure with randomness: probabilistic algorithms for constructing approximate matrix decompositions
- Finite sample approximation results for principal component analysis: A matrix perturbation approach
- scientific article; zbMATH DE number 6159604 (Why is no real title available?)
- scientific article; zbMATH DE number 3239038 (Why is no real title available?)
- Inference for Density Families Using Functional Principal Component Analysis
- Learning theory of distributed spectral algorithms
- Minimax sparse principal subspace estimation in high dimensions
- Nonasymptotic upper bounds for the reconstruction error of PCA
- On consistency and sparsity for principal components analysis in high dimensions
- On the distribution of the largest eigenvalue in principal components analysis
- Optimal principal component analysis in distributed and streaming models
- Parallelizing spectrally regularized kernel algorithms
- PCA consistency in high dimension, low sample size context
- Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices
- Practical sketching algorithms for low-rank matrix approximation
- Second order accurate distributed eigenvector computation for extremely large matrices
- Singular vector perturbation under Gaussian noise
- Sparse PCA: optimal rates and adaptive estimation
- Sub-Gaussian estimators of the mean of a random matrix with heavy-tailed entries
- The statistics and mathematics of high dimension low sample size asymptotics
Cited in
(93)- Second order accurate distributed eigenvector computation for extremely large matrices
- Distributed linear regression by averaging
- Distributed adaptive Gaussian mean estimation with unknown variance: interactive protocol helps adaptation
- Asymptotic covariance estimation by Gaussian random perturbation
- Distributed nonparametric function estimation: optimal rate of convergence and cost of adaptation
- Robust distributed estimation and variable selection for massive datasets via rank regression
- Distributed estimation in heterogeneous reduced rank regression: with application to order determination in sufficient dimension reduction
- Robust covariance estimation for distributed principal component analysis
- Adapting the Hill estimator to distributed inference: dealing with the bias
- Robust reduced rank regression in a distributed setting
- Communication-efficient distributed \(M\)-estimation with missing data
- Distributed one-step upgraded estimation for non-uniformly and non-randomly distributed data
- Distributed statistical estimation and rates of convergence in normal approximation
- Distributed estimation of principal eigenspaces
- Principal component analysis in the local differential privacy model
- Restricted Riemannian geometry for positive semidefinite matrices
- Distributed Bayesian posterior voting strategy for massive data
- WONDER: weighted one-shot distributed ridge regression in high dimensions
- Communication-efficient distributed covariance sketch, with application to distributed PCA
- scientific article; zbMATH DE number 7376764 (Why is no real title available?)
- Distributed Sufficient Dimension Reduction for Heterogeneous Massive Data
- scientific article; zbMATH DE number 7626709 (Why is no real title available?)
- scientific article; zbMATH DE number 7626713 (Why is no real title available?)
- Communication-Efficient Distributed Linear Discriminant Analysis for Binary Classification
- Distributed high-dimensional regression under a quantile loss function
- Inference for multiple heterogeneous networks with a common invariant subspace
- Communication-Efficient Distributed Eigenspace Estimation
- Distributed inference for linear support vector machine
- Optimal principal component analysis in distributed and streaming models
- A review of distributed statistical inference
- Distributed clustering using collective principal component analysis
- Robust estimation for nonrandomly distributed data
- Distributed estimation with empirical likelihood
- Distributed Sparse Composite Quantile Regression in Ultrahigh Dimensions
- A distributed community detection algorithm for large scale networks under stochastic block models
- Distributed Estimation for Principal Component Analysis: An Enlarged Eigenspace Analysis
- First-Order Newton-Type Estimator for Distributed Estimation and Inference
- Sparse online principal component analysis for parameter estimation in factor model
- A communication efficient distributed one-step estimation
- Time-varying minimum variance portfolio
- Communication-Efficient Accurate Statistical Estimation
- Two-stage communication-efficient distributed sparse M-estimation with missing data
- Distributed statistical optimization for non-randomly stored big data with application to penalized learning
- An Asymptotic Analysis of Random Partition Based Minibatch Momentum Methods for Linear Regression Models
- Meta Clustering for Collaborative Learning
- Divide-and-Conquer: A Distributed Hierarchical Factor Approach to Modeling Large-Scale Time Series Data
- Distributed estimation and inference for spatial autoregression model with large scale networks
- Distributed SGD in overparametrized linear regression
- A covariance-free iterative algorithm for distributed principal component analysis on vertically partitioned data
- Adaptive distributed support vector regression of massive data
- Distributed estimation and inference for semiparametric binary response models
- A selective review on statistical methods for massive data computation: distributed computing, subsampling, and minibatch techniques
- A Note on Distributed Quantile Regression by Pilot Sampling and One-Step Updating
- Feature Screening for Massive Data Analysis by Subsampling
- Distributed parameter estimation framework based on moment method
- Incorporating auxiliary information for improved statistical inference and its extensions to distributed algorithms with an application to personal credit
- Distributed mean dimension reduction through semi-parametric approaches
- Distributed robust estimation and inference with contaminated data
- Hypothesis testing of one sample mean vector in distributed frameworks
- Distributed estimation of spiked eigenvalues in spiked population models
- Robust two-way dimension reduction by Grassmannian barycenter
- Identifying the Structure of High-Dimensional Time Series via Eigen-Analysis
- Distributed Tensor Principal Component Analysis with Data Heterogeneity
- Communication-Efficient Distributed Sparse Learning with Oracle Property and Geometric Convergence
- Communication-efficient distributed estimation and computation using skew-normal distribution
- Distributed loading estimation for general factor model
- Privacy-preserving communication-efficient spectral clustering for distributed multiple networks
- Communication-Efficient Distributed Estimation and Inference for Cox’s Model
- Privacy-preserving estimation for non-randomly distributed data
- Limit results for estimation of connectivity matrix in multi-layer stochastic block models
- An Improved Divide-and-Conquer Approach to Estimating Mean Functional, with Application to Average Treatment Effect Estimation
- Distributed Estimation of Principal Support Vector Machines for Sufficient Dimension Reduction
- Distributed empirical likelihood inference with or without Byzantine failures
- Communication-efficient distributed gradient descent via random projection
- Efficient distributed transfer learning for large-scale Gaussian graphic models
- A Simple Divide-and-Conquer-based Distributed Method for the Accelerated Failure Time Model
- Distributed Learning for Principal Eigenspaces without Moment Constraints
- Class-Distributed Learning for Multinomial Logistic Regression with High Dimensional Features and a Large Number of Classes
- Distributed estimation for large-scale expectile regression
- Quadratic discriminant analysis in distributed frameworks
- \texttt{FedPower}: privacy-preserving distributed eigenspace estimation
- Distributed Mallows model averaging for ridge regressions
- Composite quantile regression for a distributed system with non-randomly distributed data
- Optimal subset selection for distributed local principal component analysis
- Dimension Reduction for Large-Scale Federated Data: Statistical Rate and Asymptotic Inference
- Communication-Efficient Pilot Estimation for Non-Randomly Distributed Data in Diverging Dimensions
- A Generalized Mean Approach for Distributed-PCA
- A novel approach of empirical likelihood with massive data
- Decentralized sparse linear regression via gradient-tracking
- Penalized empirical likelihood over decentralized networks
- An adaptive-to-model distributed hybrid test for conditional independence
- LEARNER: a transfer learning method for low-rank matrix estimation
- Distributed Nonparametric Regression with Heterogeneity Through Prediction-Based Aggregation
This page was built for publication: Distributed estimation of principal eigenspaces
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2284361)
