Data fusion using factor analysis and low-rank matrix completion

DOI10.1007/S11222-021-10033-7MaRDI QIDQ2058799zbMATH OpenOpenAlexFDO

Authors Daniel Ahfock, Saumyadipta Pyne, Geoffrey J. McLachlan

Publication date 9 December 2021

Published in Statistics and Computing (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2104.02888

zbMATH Keywords

factor analysisdata fusionlow-rank matrix completionstatistical file-matching

Mathematics Subject Classification ID

Computational methods for problems pertaining to statistics (62-08) Factor analysis and principal components; correspondence analysis (62H25) Numerical methods for low-rank matrix approximation; matrix compression (65F55)

Abstract: Data-fusion involves the integration of multiple related datasets. The statistical file-matching problem is a canonical data-fusion problem in multivariate analysis, where the objective is to characterise the joint distribution of a set of variables when only strict subsets of marginal distributions have been observed. Estimation of the covariance matrix of the full set of variables is challenging given the missing-data pattern. Factor analysis models use lower-dimensional latent variables in the data-generating process, and this introduces low-rank components in the complete-data matrix and the population covariance matrix. The low-rank structure of the factor analysis model can be exploited to estimate the full covariance matrix from incomplete data via low-rank matrix completion. We prove the identifiability of the factor analysis model in the statistical file-matching problem under conditions on the number of factors and the number of shared variables over the observed marginal subsets. Additionally, we provide an EM algorithm for parameter estimation. On several real datasets, the factor model gives smaller reconstruction errors in file-matching problems than the common approaches for low-rank matrix completion.

Recommendations

• Statistical matching. A frequentist theory, practical applications, and alternative Bayesian approaches
• Imputation in Data Fusion of Heterogeneous Data Sets A Model-Based Numerical Experiment
• Bayesian finite population imputation for data fusion
• Flexible low-rank statistical modeling with missing data and side information
• scientific article; zbMATH DE number 1785988

Cites work

• scientific article; zbMATH DE number 3117083 (Why is no real title available?)
• scientific article; zbMATH DE number 3567782 (Why is no real title available?)
• scientific article; zbMATH DE number 1834445 (Why is no real title available?)
• A generalized solution of the orthogonal Procrustes problem
• Asymptotically distribution‐free methods for the analysis of covariance structures
• Bayesian inference for partially identified models. Exploring the limits of limited data
• EM algorithms for ML factor analysis
• Estimating the dimension of a model
• Flexible imputation of missing data
• Generic global identification in factor analysis
• Identifiability of factor analysis: Some results and open problems
• Incorporating covariates into integrated factor analysis of multi-view data
• Integrative factorization of bidimensionally linked matrices
• Linked matrix factorization
• Model selection criteria for missing-data problems using the EM algorithm
• Nuclear-norm penalization and optimal rates for noisy low-rank matrix completion
• Partial identification in the statistical matching problem
• Spectral regularization algorithms for learning large incomplete matrices
• Statistical Matching
• Statistical matching. A frequentist theory, practical applications, and alternative Bayesian approaches
• Uncertainty analysis for statistical matching of ordered categorical variables

Cited in

(1)

• Manifold matching: joint optimization of fidelity and commensurability

This page was built for publication: Data fusion using factor analysis and low-rank matrix completion

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2058799)