Unique entity estimation with application to the Syrian conflict

DOI10.1214/18-AOAS1163zbMATH Open1405.62238arXiv1710.02690MaRDI QIDQ1624822FDOQ1624822

Rebecca C. Steorts, Beidi Chen, Anshumali Shrivastava

Publication date: 16 November 2018

Published in: The Annals of Applied Statistics (Search for Journal in Brave)

Abstract: Entity resolution identifies and removes duplicate entities in large, noisy databases and has grown in both usage and new developments as a result of increased data availability. Nevertheless, entity resolution has tradeoffs regarding assumptions of the data generation process, error rates, and computational scalability that make it a difficult task for real applications. In this paper, we focus on a related problem of unique entity estimation, which is the task of estimating the unique number of entities and associated standard errors in a data set with duplicate entities. Unique entity estimation shares many fundamental challenges of entity resolution, namely, that the computational cost of all-to-all entity comparisons is intractable for large databases. To circumvent this computational barrier, we propose an efficient (near-linear time) estimation algorithm based on locality sensitive hashing. Our estimator, under realistic assumptions, is unbiased and has provably low variance compared to existing random sampling based approaches. In addition, we empirically show its superiority over the state-of-the-art estimators on three real applications. The motivation for our work is to derive an accurate estimate of the documented, identifiable deaths in the ongoing Syrian conflict. Our methodology, when applied to the Syrian data set, provides an estimate of $191,874pm1772$ documented, identifiable deaths, which is very close to the Human Rights Data Analysis Group (HRDAG) estimate of 191,369. Our work provides an example of challenges and efforts involved in solving a real, noisy challenging problem where modeling assumptions may not hold.

Full work available at URL: https://arxiv.org/abs/1710.02690

zbMATH Keywords

clusteringhashingentity resolutionSyrian conflict

Mathematics Subject Classification ID

Classification and discrimination; cluster analysis (statistical aspects) (62H30) Applications of statistics to social sciences (62P25)

Cites Work

• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• The Complexity of Counting Cuts and of Computing the Probability that a Graph is Connected
• Detecting duplicates in a homicide registry using a Bayesian partitioning approach
• Entity resolution with empirically motivated priors
• A Bayesian Procedure for File Linking to Analyze End-of-Life Medical Costs
• A hierarchical Bayesian approach to record linkage and population size problems
• Approximating the Minimum Spanning Tree Weight in Sublinear Time
• On the Problem of Matching Lists by Samples

Cited In (6)

• A Bayesian record linkage model incorporating relational data
• Bayesian analysis of immigration in Europe with generalized logistic regression
• Bayesian hidden Markov models for latent variable labeling assignments in conflict research: application to the role ceasefires play in conflict dynamics
• Estimating the number of connected components in a graph via subgraph sampling
• A unified framework for de-duplication and population size estimation (with discussion)
• Improving Probabilistic Record Linkage Using Statistical Prediction Models

Recommendations

• Entity resolution with empirically motivated priors 👍 👎
• Evaluating entity-description conflict on duplicated data 👍 👎
• Entity resolution for probabilistic data 👍 👎
• Singular sources mining using evidential conflict analysis 👍 👎
• Title not available (Why is that?) 👍 👎

This page was built for publication: Unique entity estimation with application to the Syrian conflict

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q1624822)