Main effects and interactions in mixed and incomplete data frames

DOI10.1080/01621459.2019.1623041zbMATH Open1441.62145arXiv1806.09734OpenAlexW2963603703WikidataQ127804282 ScholiaQ127804282MaRDI QIDQ146484FDOQ146484

Authors: Geneviève Robin, Olga Klopp, Julie Josse, Éric Moulines, Robert Tibshirani, Geneviève Robin, Olga Klopp, Julie Josse, Robert Tibshirani, Eric Moulines 

Publication date: 26 June 2018

Published in: Journal of the American Statistical Association (Search for Journal in Brave)

Abstract: A mixed data frame (MDF) is a table collecting categorical, numerical and count observations. The use of MDF is widespread in statistics and the applications are numerous from abundance data in ecology to recommender systems. In many cases, an MDF exhibits simultaneously main effects, such as row, column or group effects and interactions, for which a low-rank model has often been suggested. Although the literature on low-rank approximations is very substantial, with few exceptions, existing methods do not allow to incorporate main effects and interactions while providing statistical guarantees. The present work fills this gap. We propose an estimation method which allows to recover simultaneously the main effects and the interactions. We show that our method is near optimal under conditions which are met in our targeted applications. We also propose an optimization algorithm which provably converges to an optimal solution. Numerical experiments reveal that our method, mimi, performs well when the main effects are sparse and the interaction matrix has low-rank. We also show that mimi compares favorably to existing methods, in particular when the main effects are significantly large compared to the interactions, and when the proportion of missing entries is large. The method is available as an R package on the Comprehensive R Archive Network.

Full work available at URL: https://arxiv.org/abs/1806.09734

Recommendations

• A principal component method to impute missing values for mixed data
• Pairwise sparse + low-rank models for variables of mixed type
• scientific article; zbMATH DE number 3911503
• Missing-values adjustment for mixed-type data
• Low-rank model with covariates for count data with missing values

zbMATH Keywords

missing valuesheterogeneous datalow-rank matrix completion

Mathematics Subject Classification ID

Statistical aspects of big data and data science (62R07) Estimation in multivariate analysis (62H12)

Cites Work

• Spectral regularization algorithms for learning large incomplete matrices
• Matrix completion and low-rank SVD via fast alternating least squares
• Generalized low rank models
• Statistics for high-dimensional data. Methods, theory and applications.
• Title not available (Why is that?)
• Title not available (Why is that?)
• Robust principal component analysis?
• Flexible low-rank statistical modeling with missing data and side information
• A coordinate gradient descent method for nonsmooth separable minimization
• Simple structure in component analysis techniques for mixtures of qualitative and quantitative variables
• A Max-Norm Constrained Minimization Approach to 1-Bit Matrix Completion
• Rank-Sparsity Incoherence for Matrix Decomposition
• Imputation of Mixed Data With Multilevel Singular Value Decomposition
• Literature survey on low rank approximation of matrices
• Poisson Matrix Recovery and Completion
• Multiple factor analysis by example using R
• Matrix completion with covariate information
• Robust Matrix Decomposition With Sparse Corruptions
• Recovery of Low-Rank Plus Compressed Sparse Matrices With Application to Unveiling Traffic Anomalies
• Modeling item-item similarities for personalized recommendations on Yahoo! front page
• Robust matrix completion
• Adaptive multinomial matrix completion

Cited In (7)

• An adaptation for iterative structured matrix completion
• Matrix completion under complex survey sampling
• Generalized Low-Rank Plus Sparse Tensor Estimation by Fast Riemannian Optimization
• Safety signal detection with control of latent factors
• Imputation and low-rank estimation with missing not at random data
• Detecting arrays for main effects
• mimi

Uses Software

• R
• glmnet
• missMDA
• MICE
• softImpute
• LowRankModels
• Pyglrm

This page was built for publication: Main effects and interactions in mixed and incomplete data frames

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q146484)