Data fusion using factor analysis and low-rank matrix completion

DOI10.1007/S11222-021-10033-7zbMATH Open1475.62014arXiv2104.02888OpenAlexW3187574706MaRDI QIDQ2058799FDOQ2058799

Authors: Daniel Ahfock, Saumyadipta Pyne, Geoffrey J. McLachlan 

Publication date: 9 December 2021

Published in: Statistics and Computing (Search for Journal in Brave)

Abstract: Data-fusion involves the integration of multiple related datasets. The statistical file-matching problem is a canonical data-fusion problem in multivariate analysis, where the objective is to characterise the joint distribution of a set of variables when only strict subsets of marginal distributions have been observed. Estimation of the covariance matrix of the full set of variables is challenging given the missing-data pattern. Factor analysis models use lower-dimensional latent variables in the data-generating process, and this introduces low-rank components in the complete-data matrix and the population covariance matrix. The low-rank structure of the factor analysis model can be exploited to estimate the full covariance matrix from incomplete data via low-rank matrix completion. We prove the identifiability of the factor analysis model in the statistical file-matching problem under conditions on the number of factors and the number of shared variables over the observed marginal subsets. Additionally, we provide an EM algorithm for parameter estimation. On several real datasets, the factor model gives smaller reconstruction errors in file-matching problems than the common approaches for low-rank matrix completion.

Full work available at URL: https://arxiv.org/abs/2104.02888

Recommendations

• Statistical matching. A frequentist theory, practical applications, and alternative Bayesian approaches
• Imputation in Data Fusion of Heterogeneous Data Sets A Model-Based Numerical Experiment
• Bayesian finite population imputation for data fusion
• Flexible low-rank statistical modeling with missing data and side information
• scientific article; zbMATH DE number 1785988

zbMATH Keywords

factor analysisdata fusionlow-rank matrix completionstatistical file-matching

Mathematics Subject Classification ID

Computational methods for problems pertaining to statistics (62-08) Factor analysis and principal components; correspondence analysis (62H25) Numerical methods for low-rank matrix approximation; matrix compression (65F55)

Cites Work

• Spectral regularization algorithms for learning large incomplete matrices
• Estimating the dimension of a model
• Title not available (Why is that?)
• Title not available (Why is that?)
• Model selection criteria for missing-data problems using the EM algorithm
• Statistical Matching
• Nuclear-norm penalization and optimal rates for noisy low-rank matrix completion
• Generic global identification in factor analysis
• A generalized solution of the orthogonal Procrustes problem
• Title not available (Why is that?)
• Flexible imputation of missing data
• EM algorithms for ML factor analysis
• Asymptotically distribution‐free methods for the analysis of covariance structures
• Statistical matching. A frequentist theory, practical applications, and alternative Bayesian approaches
• Identifiability of factor analysis: Some results and open problems
• Bayesian inference for partially identified models. Exploring the limits of limited data
• Uncertainty analysis for statistical matching of ordered categorical variables
• Partial identification in the statistical matching problem
• Incorporating covariates into integrated factor analysis of multi-view data
• Integrative factorization of bidimensionally linked matrices
• Linked matrix factorization

Cited In (1)

• Manifold matching: joint optimization of fidelity and commensurability

This page was built for publication: Data fusion using factor analysis and low-rank matrix completion

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2058799)