Multifile Partitioning for Record Linkage and Duplicate Detection

DOI10.1080/01621459.2021.2013242MaRDI QIDQ6077572zbMATH OpenFDO

Authors Mauricio Sadinle

Publication date 18 October 2023

Published in Journal of the American Statistical Association (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2110.03839

zbMATH Keywords

data matchingdata mergingentity resolutionmicroclustering

Mathematics Subject Classification ID

Statistics (62-XX)

Abstract: Merging datafiles containing information on overlapping sets of entities is a challenging task in the absence of unique identifiers, and is further complicated when some entities are duplicated in the datafiles. Most approaches to this problem have focused on linking two files assumed to be free of duplicates, or on detecting which records in a single file are duplicates. However, it is common in practice to encounter scenarios that fit somewhere in between or beyond these two settings. We propose a Bayesian approach for the general setting of multifile record linkage and duplicate detection. We use a novel partition representation to propose a structured prior for partitions that can incorporate prior information about the data collection processes of the datafiles in a flexible manner, and extend previous models for comparison data to accommodate the multifile setting. We also introduce a family of loss functions to derive Bayes estimates of partitions that allow uncertain portions of the partitions to be left unresolved. The performance of our proposed methodology is explored through extensive simulations. Code implementing the methodology is available at https://github.com/aleshing/multilink .

Recommendations

• scientific article; zbMATH DE number 2015502
• Efficient detection of discrepancies in multiple file copies
• Optimal partitioning of data chunks in deduplication systems
• Multi-attribute hashing with multiple file copies for high performance partial-match retrieval

Cites work

• A Generalized Fellegi–Sunter Framework for Multiple Record Linkage With Application to Homicide Record Systems
• A hierarchical Bayesian approach to record linkage and population size problems
• A unified framework for de-duplication and population size estimation (with discussion)
• Bayesian cluster analysis
• Bayesian cluster analysis: point estimation and credible balls (with discussion)
• Classification with reject option
• Comparing clusterings -- an information based distance
• d-blink: Distributed End-to-End Bayesian Entity Resolution
• Detecting duplicates in a homicide registry using a Bayesian partitioning approach
• Entity resolution with empirically motivated priors
• scientific article; zbMATH DE number 3297798 (Why is no real title available?)
• Random Partition Models for Microclustering Tasks

Cited in

(6)

• A Practical Approach to Proper Inference with Linked Data
• Generative Filtering for Recursive Bayesian Inference with Streaming Data
• Analysis of linked files: a missing data perspective
• Efficient and scalable bipartite matching with fast beta linkage (fabl)
• Fast Bayesian Record Linkage for Streaming Data Contexts
• A review of modern multinomial-derived and partition-based record-linkage methods

This page was built for publication: Multifile Partitioning for Record Linkage and Duplicate Detection

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6077572)