Will Lyon- Entity Resolution
4 likes2,094 views
The document discusses an active learning approach to entity resolution in graph data, highlighting techniques such as inferred relationships, dataset joining, and aggregation. It emphasizes the importance of distinguishing between similar records to address deduplication concerns, utilizing methods like edit distance and logistic regression for quantifying similarities. The author provides code samples and resources for implementing entity resolution using Python and Neo4j.
![(you)-[:HAVE]->(?)
(?)<-[:ANSWERS]-(will)](https://image.slidesharecdn.com/willlyontalk-entityresolution-170914222848/85/Will-Lyon-Entity-Resolution-66-320.jpg)
Will Lyon- Entity Resolution
- 1. Entity Resolution In Graph Data An Active Learning Approach William Lyon @lyonwj lyonwj.com Sept 2017
- 2. William Lyon Developer Relations Engineer @neo4j will@neo4j.com @lyonwj lyonwj.com
- 11. contributions Synthetic id on concat(name, zip) https://github.com/johnymontana/neo4j-datasets/tree/master/us-fec-elections-2016
- 12. Sample data
- 16. Entity Resolution In Graph Data ?
- 17. Entity Resolution In Graph Data 1) Inferred relationships 2) Joining datasets 3) Aggregating traversals
- 20. 1) Inferred relationships LIVES_WITH https://offshoreleaks.icij.org/pages/database
- 27. Entity Resolution An Active Learning Approach
- 28. Are these the same entity? Name Address Phone Email Bob Loblaw 111 E 5th Ave. San Mateo, CA bob@neo4j.com Name Address Phone Email Robert Loblaw 111 5th Ave 855-636-4532 bob@neo4j.com
- 29. Are these the same entity? Name Address Phone Email Bob Loblaw 111 E 5th Ave. San Mateo, CA bob@neo4j.com Name Address Phone Email Robert Loblaw 111 5th Ave 855-636-4532 bob@neo4j.com Probably, but how to quantify?
- 30. Are these the same entity? Name Bob Loblaw Name Robert Loblaw Probably, but how to quantify?
- 31. Are these the same entity? Name Bob Loblaw Name Robert Loblaw String distance (similarity) metric
- 32. Are these the same entity? Name Bob Loblaw Name Robert Loblaw Edit distance
- 33. Are these the same entity? Name Bob Loblaw Name Robert Loblaw Edit distance Edits required to convert “Bob Loblaw” to “Robert Loblaw”
- 34. Are these the same entity? Name Bob Loblaw Name Robert Loblaw Edit distance 4
- 35. Are these the same entity? Name Bob Loblaw Name Robert Loblaw Edit distance
- 36. Are these the same entity? Name Bob Loblaw Name Robert Loblaw Edit distance
- 37. Are these the same entity? Name Bob Loblaw Name Robert Loblaw TF/IDF • Term based • set of words • Order doesn’t matter • Words weighted based on probability
- 38. Are these the same entity? Name Bob Loblaw Name Robert Loblaw TF/IDF • Pro: • takes advantage of frequency • Order doesn’t matter • Willam Cohen -vs- Cohen, William • Con: • Spelling errors / abbreviations • Order doesn’t matter • City National Bank -vs- National City Bank
- 39. Are these the same entity? Name Bob Loblaw Name Robert Loblaw Edit distance probabilistic extensions • Gap distance • Edit distance + HMM
- 40. Are these the same entity? Name Bob Loblaw Name Robert Loblaw Soundex • Phonetic indexing scheme • Genealogy • Soundex code • Hash
- 41. Are these the same entity? Name Bob Loblaw Name Robert Loblaw Soundex
- 42. Are these the same entity? Name Address Phone Email Bob Loblaw 111 E 5th Ave. San Mateo, CA bob@neo4j.com Name Address Phone Email Robert Loblaw 111 5th Ave 855-636-4532 bob@neo4j.com Dist 4
- 43. Are these the same entity? Name Address Phone Email Bob Loblaw 111 E 5th Ave. San Mateo, CA bob@neo4j.com Name Address Phone Email Robert Loblaw 111 5th Ave 855-636-4532 bob@neo4j.com Dist 4 17 Null 0
- 44. Are these the same entity? Name Address Phone Email Bob Loblaw 111 E 5th Ave. San Mateo, CA bob@neo4j.com Name Address Phone Email Robert Loblaw 111 5th Ave 855-636-4532 bob@neo4j.com Dist 4 17 Null 0 Weight ??? ??? ??? ???
- 45. Are these the same entity? Name Address Phone Email Bob Loblaw 111 E 5th Ave. San Mateo, CA bob@neo4j.com Name Address Phone Email Robert Loblaw 111 5th Ave 855-636-4532 bob@neo4j.com Dist 4 17 Null 0 Weight 0.2 0.03 0.02 0.75
- 46. Are these the same entity? Name Address Phone Email Bob Loblaw 111 E 5th Ave. San Mateo, CA bob@neo4j.com Name Address Phone Email Robert Loblaw 111 5th Ave 855-636-4532 bob@neo4j.com Dist 4 17 Null 0 Weight 0.2 0.03 0.02 0.75 Weighted distance: (4*0.2)+(0.03*17)+(0*0.02)+(0*.75) = 1.31
- 47. Are these the same entity? Name Address Phone Email Bob Loblaw 111 E 5th Ave. San Mateo, CA bob@neo4j.com Name Address Phone Email Robert Loblaw 111 5th Ave 855-636-4532 bob@neo4j.com Dist 4 17 Null 0 Weight 0.2 0.03 0.02 0.75 Where did the weights come from?
- 48. Active Learning
- 49. Active Learning • Goal: learn weights for distance per field • Example pairs w/ labels • Minimize human labeling time • Present borderline pairs to human for labeling • Relearn weights • Iterate Name Address Phone Email Bob Loblaw 111 E 5th Ave. San Mateo, CA bob@neo4j.com Name Address Phone Email Robert Loblaw 111 5th Ave 855-636-4532 bob@neo4j.com
- 50. Logistic regression • Categorical dependent variable • binary (0,1) • Classification • Fit logistic function https://en.wikipedia.org/wiki/Logistic_function
- 51. Logistic regression - example Distance 0.50 0.75 1.00 1.25 1.50 1.75 1.75 2.00 2.25 2.50 2.75 3.00 3.25 3.50 4.00 4.25 4.50 4.75 5.00 5.50 Nonduplicate 0 0 0 0 0 0 1 0 1 0 1 0 1 0 1 1 1 1 1 1
- 52. Logistic regression - example Probability of non-duplicate vs edit distance Probabilityofnon-duplicate Edit distance Distance 0.50 0.75 1.00 1.25 1.50 1.75 1.75 2.00 2.25 2.50 2.75 3.00 3.25 3.50 4.00 4.25 4.50 4.75 5.00 5.50 Nonduplicate 0 0 0 0 0 0 1 0 1 0 1 0 1 0 1 1 1 1 1 1
- 53. Logistic regression - example Probability of non-duplicate vs edit distance Probabilityofnon-duplicate Edit distance Distance 0.50 0.75 1.00 1.25 1.50 1.75 1.75 2.00 2.25 2.50 2.75 3.00 3.25 3.50 4.00 4.25 4.50 4.75 5.00 5.50 Nonduplicate 0 0 0 0 0 0 1 0 1 0 1 0 1 0 1 1 1 1 1 1 Coefficient Std Err Intercept -4.0777 1.7610 Distance 1.5046 0.6287
- 54. Blocking - making smart comparisons • Problem: All-pairs comparison is expensive • (1000*999) / 2 = 499,500 pairs • Duplicates are rare • almost all pairs are not duplicates • Blocking functions limit record pairs to be compared • “Somewhat similar” records
- 55. Blocking - making smart comparisons • Predicate blocks • whole field • token field • common integer • same three char start • common n gram • Index block • inverted index • find similar records close to each other in the index
- 56. Code samples
- 57. Dedupe by Datamade • Entity resolution Python library • Active learning • Logistic regression • Blocking • Generate CSV or query database directly • Cypher, python driver for Neo4j https://github.com/dedupeio
- 58. Training
- 59. Active Learning
- 60. Writing results
- 62. Resources