Deduplication
- 1. DeduplicationBouvet BigOne, 2011-04-13Lars Marius Garshol, <larsga@bouvet.no>http://twitter.com/larsga
- 3. The problemThe suppliers tableReal-world data is very, very messy
- 4. The problem – take 2SuppliersCustomersCustomersCustomersCompaniesCRMBillingERPEach of these has internal duplicates,plus duplicates across the tables. Noeasy fix.
- 5. But ... what about identifiers?No, there are no system IDs across these tablesYes, there are outside identifiersorganization number for companiespersonal number for peopleBut, these are problematicmany records don't have themthey are inconsistently formattedsometimes they are misspelledsome parts of huge organizations have the same org number, but need to be treated as separate
- 6. First attempt at solutionI wrote a simple Python script in ~2 hoursIt does the following:load all recordsnormalize the datastrip extra whitespace, lowercase, remove letters from org codes...use Bayesian inferencing for matching
- 8. MatchingThis sums out to 0.93 probability
- 9. ProblemsThe functions comparing values are still pretty primitivePerformance is abysmal90 minutes to process 14,500 recordsperformance is O(n2)total number of records is ~2.5 milliontime to process all records: 1 year 10 monthsNow what?
- 10. An ideaWell, we don't necessarily need to compare each record with all others if we have indexeswe can look up the records which have matching valuesUse DBM for the indexes, for exampleUnfortunately, these only allow exact matchingBut, we can break up complex values into tokens, and index thoseHang on, isn't this rather like a search engine?Bing!Let's try Lucene!
- 11. Lucene-based prototypeI whip out Jython and try itNew script first builds Lucene indexThen searches all records against the indexTime to process 14,500 records: 1 minuteNow we're talking...
- 12. Reality sets inA splash of cold water to the face
- 13. Prior artIt turns out people have been doing this beforeThey call itentity resolutionidentity resolutionmerge/purgededuplicationrecord linkage...This makes Googling for information an absolute nightmare
- 14. Existing toolsSeveral commercial toolsthey look big and expensive: we skip thoseStian found some open source toolsOyster: slow, bad architecture, primitive matchingSERF: slow, bad architectureI’ve later found more, but was not impressedSo, it seems we still have to do it ourselves
- 15. Finds in the research literatureGeneralproblem is well-understood"naïve Bayes" is naïvelots of interesting work on value comparisonsperformance problem 'solved' with "blocking"build a key from parts of the datasort records by keycompare each record with m nearest neighboursperformance goes from O(n2) to O(n m)parallel processing widely usedSwoosh papercompare and merge should have ICAR1 propertiesoptimal algorithms for general merge foundrun-time for 14,000 records ~1.5 hours...1 Idempotence, commutativity, associativity, reflexivity
- 16. Good research papersThreat and Fraud Intelligence, Las Vegas Style, Jeff Jonashttp://jeffjonas.typepad.com/IEEE.Identity.Resolution.pdfReal-world data is dirty: Data Cleansing and the Merge/Purge Problem, Hernandez & Stolfohttp://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.30.3496&rep=rep1&type=pdfSwoosh: a generic approach to entity resolution, Benjelloun, Garcia-Molina et alhttp://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.122.5696&rep=rep1&type=pdf
- 18. Java deduplication engineWork in progressso far spent only ~20 hours on itonly command-line batch client built so farBased on Lucene 3.1Open source (on Google Code)http://code.google.com/p/duke/Blazingly fast960,000 records in 11 minutes on this laptop
- 19. Architecturedata inequivalences outSDshare clientSDshare serverRDF frontendDatastore APIDuke engineLuceneH2 database
- 20. Architecture #2data inlink file outCommand-line clientMore frontends: JDBC
- 21. SPARQL
- 22. RDF file
- 23. ...CSV frontendDatastore APIDuke engineLucene
- 24. Architecture #3data inequivalences outREST interfaceX frontendDatastore APIDuke engineLuceneH2 database
- 25. WeaknessesTied to naïve Bayes modelresearch shows more sophisticated models perform betternon-trivial to reconcile these with index lookupValue comparison sophistication limitedLucene does support Levenshtein queries(these are slow, though. will be fast in 4.x)