Unique entity estimation with application to the Syrian conflict

DOI10.1214/18-AOAS1163MaRDI QIDQ1624822zbMATHFDO

Authors Beidi Chen, Anshumali Shrivastava, Rebecca C. Steorts

Publication date 16 November 2018

Published in The Annals of Applied Statistics (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1710.02690

zbMATH Keywords

clusteringhashingentity resolutionSyrian conflict

Mathematics Subject Classification ID

Classification and discrimination; cluster analysis (statistical aspects) (62H30) Applications of statistics to social sciences (62P25)

Abstract: Entity resolution identifies and removes duplicate entities in large, noisy databases and has grown in both usage and new developments as a result of increased data availability. Nevertheless, entity resolution has tradeoffs regarding assumptions of the data generation process, error rates, and computational scalability that make it a difficult task for real applications. In this paper, we focus on a related problem of unique entity estimation, which is the task of estimating the unique number of entities and associated standard errors in a data set with duplicate entities. Unique entity estimation shares many fundamental challenges of entity resolution, namely, that the computational cost of all-to-all entity comparisons is intractable for large databases. To circumvent this computational barrier, we propose an efficient (near-linear time) estimation algorithm based on locality sensitive hashing. Our estimator, under realistic assumptions, is unbiased and has provably low variance compared to existing random sampling based approaches. In addition, we empirically show its superiority over the state-of-the-art estimators on three real applications. The motivation for our work is to derive an accurate estimate of the documented, identifiable deaths in the ongoing Syrian conflict. Our methodology, when applied to the Syrian data set, provides an estimate of $191,874pm1772$ documented, identifiable deaths, which is very close to the Human Rights Data Analysis Group (HRDAG) estimate of 191,369. Our work provides an example of challenges and efforts involved in solving a real, noisy challenging problem where modeling assumptions may not hold.

Recommendations

• Entity resolution for probabilistic data
• Theoretical foundations of entity resolution models
• Entity resolution with empirically motivated priors
• Evaluating entity-description conflict on duplicated data
• Singular sources mining using evidential conflict analysis

Cites work

• scientific article; zbMATH DE number 3168330 (Why is no real title available?)
• scientific article; zbMATH DE number 3606505 (Why is no real title available?)
• scientific article; zbMATH DE number 1775450 (Why is no real title available?)
• scientific article; zbMATH DE number 3297798 (Why is no real title available?)
• A Bayesian procedure for file linking to analyze end-of-life medical costs
• A hierarchical Bayesian approach to record linkage and population size problems
• Approximating the Minimum Spanning Tree Weight in Sublinear Time
• Detecting duplicates in a homicide registry using a Bayesian partitioning approach
• Entity resolution with empirically motivated priors
• On the Problem of Matching Lists by Samples
• The Complexity of Counting Cuts and of Computing the Probability that a Graph is Connected

Cited in

(9)

• A unified framework for de-duplication and population size estimation (with discussion)
• Bayesian analysis of immigration in Europe with generalized logistic regression
• Evaluating entity-description conflict on duplicated data
• A Bayesian record linkage model incorporating relational data
• Counting casualties: A framework for respectful, useful records
• Bayesian hidden Markov models for latent variable labeling assignments in conflict research: application to the role ceasefires play in conflict dynamics
• How many refugees and migrants died trying to reach Europe? Joint population size and total estimation
• Improving Probabilistic Record Linkage Using Statistical Prediction Models
• Estimating the number of connected components in a graph via subgraph sampling

This page was built for publication: Unique entity estimation with application to the Syrian conflict

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q1624822)