Indexing Techniques for Scalable Record Linkage and Deduplication
1 like1,681 views
This document discusses indexing techniques for scalable record linkage and deduplication. It introduces the problems of record linkage on large datasets that do not fit in memory and addresses corrupted data. Blocking is presented as a common approach, where similar records are grouped into blocks to reduce the number of record pairs that must be compared. The document also discusses research on developing machine learning techniques to automatically learn optimal blocking keys and blocking functions. Evaluation frameworks for record linkage are introduced. The sorted neighborhood method is described in detail, including how it creates keys, sorts data, and merges records to link them.
![Record Linkage
Record Linkage Approaches
Blocking
.[] [] [] [] Similar values.
Blocking key.
Trade-off of size: False negatives vs cost.
Blocking Keys
No. of true matches in the candidate record pairs ⇑.
Total No. of candidate pairs ⇓.
Pradeeban Kathiravelu (IST-ULisboa) Record Linkage 6 / 18](https://image.slidesharecdn.com/dqpradeebanpresentation3-150414150156-conversion-gate01/85/Indexing-Techniques-for-Scalable-Record-Linkage-and-Deduplication-6-320.jpg)
Indexing Techniques for Scalable Record Linkage and Deduplication
- 1. Indexing Techniques for Scalable Record Linkage and Deduplication Pradeeban Kathiravelu INESC-ID Lisboa Instituto Superior T´ecnico, Universidade de Lisboa Lisbon, Portugal Data Quality – Presentation 3 April 14, 2015. Pradeeban Kathiravelu (IST-ULisboa) Record Linkage 1 / 18
- 2. Introduction Introduction Matching. Approach known as: Data or Record Linkage. Data or Field Matching. The Merge/Purge Problem. Too large to fit in the main memory. Corrupted incoming new data requiring complex tests. Importance of accuracy, than missing data. Pradeeban Kathiravelu (IST-ULisboa) Record Linkage 2 / 18
- 3. Introduction Matching Records {Data|Record} Linkage | {Data|Field} Matching Pradeeban Kathiravelu (IST-ULisboa) Record Linkage 3 / 18
- 4. Introduction Motivation Linked Data Improving data quality and integrity. Allowing re-use of existing data sources. Reducing costs and efforts in data acquisition. Multiple Domains Fraud and crime detection. Pervasive health systems. Enterprise business systems. Pradeeban Kathiravelu (IST-ULisboa) Record Linkage 4 / 18
- 5. Introduction Indexing in Record Linkage Pradeeban Kathiravelu (IST-ULisboa) Record Linkage 5 / 18
- 6. Record Linkage Record Linkage Approaches Blocking .[] [] [] [] Similar values. Blocking key. Trade-off of size: False negatives vs cost. Blocking Keys No. of true matches in the candidate record pairs ⇑. Total No. of candidate pairs ⇓. Pradeeban Kathiravelu (IST-ULisboa) Record Linkage 6 / 18
- 7. Record Linkage Research Avenues Scaling to large data sets. While keeping a high linkage quality. Development of techniques that can learn optimal blocking key definitions. Manual ⇒ Supervised machine learning based approaches. Machine learning approaches leveraging, Predicate-based formulations of learnable blocking functions. The sequential covering algorithm, which discovers disjunctive sets of rules. Pradeeban Kathiravelu (IST-ULisboa) Record Linkage 7 / 18
- 8. Evaluation Evaluation Evaluation Framework Febrl (Freely Extensible Biomedical Record Linkage). Developed in Python - https://sourceforge.net/projects/febrl/ data standardisation (segmentation and cleaning). probabilistic record linkage (”fuzzy” matching) Data Sets SecondString Toolkit. Developed in Java - http://secondstring.sourceforge.net/ Approximate string-matching techniques. Census, bibliographic, restaurant, and CD records. Pradeeban Kathiravelu (IST-ULisboa) Record Linkage 8 / 18
- 9. Evaluation Indexing Techniques Pradeeban Kathiravelu (IST-ULisboa) Record Linkage 9 / 18
- 10. Sorted-Neighborhood Sorted-Neighborhood method Partition the data. Sort the partitions before the matching. with the most important BKV Corrupted keys? Approach: Create Keys Sort Data Merge Pradeeban Kathiravelu (IST-ULisboa) Record Linkage 10 / 18
- 11. Sorted-Neighborhood Case Pradeeban Kathiravelu (IST-ULisboa) Record Linkage 11 / 18
- 12. Sorted-Neighborhood Equational Theory Pradeeban Kathiravelu (IST-ULisboa) Record Linkage 12 / 18
- 13. Sorted-Neighborhood Accuracy of Sorted-Neighborhood method Pradeeban Kathiravelu (IST-ULisboa) Record Linkage 13 / 18
- 14. Sorted-Neighborhood Clustering Methods vs. SNM Pradeeban Kathiravelu (IST-ULisboa) Record Linkage 14 / 18
- 15. Sorted-Neighborhood Memory-based database (13751 records) Pradeeban Kathiravelu (IST-ULisboa) Record Linkage 15 / 18
- 16. Sorted-Neighborhood Multiple Processors (1 million records; width = 10) Pradeeban Kathiravelu (IST-ULisboa) Record Linkage 16 / 18
- 17. Sorted-Neighborhood Time Performance Pradeeban Kathiravelu (IST-ULisboa) Record Linkage 17 / 18
- 18. Sorted-Neighborhood References Christen, P. (2012). A survey of indexing techniques for scalable record linkage and deduplication. Knowledge and Data Engineering, IEEE Transactions on, 24(9), 1537-1555. Hern´andez, M. A., & Stolfo, S. J. (1995, June). The merge/purge problem for large databases. In ACM SIGMOD Record (Vol. 24, No. 2, pp. 127-138). ACM. Thank you! Pradeeban Kathiravelu (IST-ULisboa) Record Linkage 18 / 18