Entity Matching for Semistructured Data in the Cloud
3 likes874 views
The document discusses entity matching for semistructured data in the cloud. It presents ChuQL, an entity matching architecture, and MAXIM, an entity matching system implemented in a Hadoop cluster. MAXIM uses a three stage process - preparation, blocking, and matching. The preparation stage extracts and indexes data. The blocking stage generates candidate record pairs. The matching stage applies similarity functions to candidate pairs.
![Motivation
Problem Statement
Definition
Given two datasets of records, R and S, a set of attributes
a1 , . . . , an , a set of similarity functions sima1 , . . . , siman and a
similarity threshold τ , the task between R and S is defined as
finding and combining all pairs of records from R and S where
n
i=1 simai (R.ai , S.ai ) ≥ τ
{{Cite book
{{Cite book
| last = Mumford
| last = Mumford
| first = David
| first = David <record id=”6627383”>
<record id=”6627383”>
| authorlink = David Mumford
| authorlink = David Mumford <author>David Mumford</author>
<author>David Mumford</author>
| title = The Red Book of Varieties and Schemes
| title = The Red Book of Varieties and Schemes <title>The red book of Varieties and
<title>The red book of Varieties and
| publisher = [[Springer]]
| publisher = [[Springer]] Schemes</title>
Schemes</title>
| location = Berlin
| location = Berlin <publisher>Springer</publisher>
<publisher>Springer</publisher>
| date = 1999
| date = 1999 <year>1999</year>
<year>1999</year>
| page = 198
| page = 198 <doi>10.1007/b62130</doi>
<doi>10.1007/b62130</doi>
| doi = 10.1007/b62130
| doi = 10.1007/b62130 </record>
</record>
| isbn = 354063293X
| isbn = 354063293X
}}
}}
Wikipedia Data set CiteSeer Data set
Marcus Paradies Entity Matching for Semistructured Data in the Cloud
5 / 19](https://image.slidesharecdn.com/sac-2012-120330082005-phpapp01/85/Entity-Matching-for-Semistructured-Data-in-the-Cloud-8-320.jpg)
![Motivation
Problem Statement
Definition
Given two datasets of records, R and S, a set of attributes
a1 , . . . , an , a set of similarity functions sima1 , . . . , siman and a
similarity threshold τ , the task between R and S is defined as
finding and combining all pairs of records from R and S where
n
i=1 simai (R.ai , S.ai ) ≥ τ
{{Cite book
{{Cite book
| last = Mumford
| last = Mumford
| first = David
| first = David <record id=”6627383”>
<record id=”6627383”>
| authorlink = David Mumford
| authorlink = David Mumford <author>David Mumford</author>
<author>David Mumford</author>
| title = The Red Book of Varieties and Schemes
| title = The Red Book of Varieties and Schemes <title>The red book of Varieties and
<title>The red book of Varieties and
| publisher = [[Springer]]
| publisher = [[Springer]] Schemes</title>
Schemes</title>
| location = Berlin
| location = Berlin <publisher>Springer</publisher>
<publisher>Springer</publisher>
| date = 1999
| date = 1999 <year>1999</year>
<year>1999</year>
| page = 198
| page = 198 <doi>10.1007/b62130</doi>
<doi>10.1007/b62130</doi>
| doi = 10.1007/b62130
| doi = 10.1007/b62130 </record>
</record>
| isbn = 354063293X
| isbn = 354063293X
}}
}}
Wikipedia Data set CiteSeer Data set
Marcus Paradies Entity Matching for Semistructured Data in the Cloud
5 / 19](https://image.slidesharecdn.com/sac-2012-120330082005-phpapp01/85/Entity-Matching-for-Semistructured-Data-in-the-Cloud-9-320.jpg)
Entity Matching for Semistructured Data in the Cloud
- 1. Entity Matching for Semistructured Data in the Cloud Marcus Paradies ACM SAC 2012 - CC Track March 27, 2012 Marcus Paradies Entity Matching for Semistructured Data in the Cloud 1 / 19
- 2. Outline 1 Motivation 2 ChuQL 3 Entity Matching 4 MAXIM: Entity Matching in the Cloud 5 Summary Marcus Paradies Entity Matching for Semistructured Data in the Cloud 2 / 19
- 3. Motivation Enriching/Improving Wikipedia References from Wikipedia article Hash join Marcus Paradies Entity Matching for Semistructured Data in the Cloud 3 / 19
- 4. Motivation Enriching/Improving Wikipedia Lookup in the CiteSeer database Marcus Paradies Entity Matching for Semistructured Data in the Cloud 3 / 19
- 5. Motivation Enriching/Improving Wikipedia Lookup in Google Marcus Paradies Entity Matching for Semistructured Data in the Cloud 3 / 19
- 6. Motivation Wikipedia in a nutshell Characteristics 3.7 Mio articles (english Wikipedia database) Dataset size about 30GB of XML (without history) 3.6 Mio references References are categorized into books, journals, websites, etc. Marcus Paradies Entity Matching for Semistructured Data in the Cloud 4 / 19
- 7. Motivation Wikipedia in a nutshell Characteristics 3.7 Mio articles (english Wikipedia database) Dataset size about 30GB of XML (without history) 3.6 Mio references References are categorized into books, journals, websites, etc. Challenges Articles in Wikipedia are incomplete Articles in Wikipedia are inaccurate Articles in Wikipedia are subjective Marcus Paradies Entity Matching for Semistructured Data in the Cloud 4 / 19
- 8. Motivation Problem Statement Definition Given two datasets of records, R and S, a set of attributes a1 , . . . , an , a set of similarity functions sima1 , . . . , siman and a similarity threshold τ , the task between R and S is defined as finding and combining all pairs of records from R and S where n i=1 simai (R.ai , S.ai ) ≥ τ {{Cite book {{Cite book | last = Mumford | last = Mumford | first = David | first = David <record id=”6627383”> <record id=”6627383”> | authorlink = David Mumford | authorlink = David Mumford <author>David Mumford</author> <author>David Mumford</author> | title = The Red Book of Varieties and Schemes | title = The Red Book of Varieties and Schemes <title>The red book of Varieties and <title>The red book of Varieties and | publisher = [[Springer]] | publisher = [[Springer]] Schemes</title> Schemes</title> | location = Berlin | location = Berlin <publisher>Springer</publisher> <publisher>Springer</publisher> | date = 1999 | date = 1999 <year>1999</year> <year>1999</year> | page = 198 | page = 198 <doi>10.1007/b62130</doi> <doi>10.1007/b62130</doi> | doi = 10.1007/b62130 | doi = 10.1007/b62130 </record> </record> | isbn = 354063293X | isbn = 354063293X }} }} Wikipedia Data set CiteSeer Data set Marcus Paradies Entity Matching for Semistructured Data in the Cloud 5 / 19
- 9. Motivation Problem Statement Definition Given two datasets of records, R and S, a set of attributes a1 , . . . , an , a set of similarity functions sima1 , . . . , siman and a similarity threshold τ , the task between R and S is defined as finding and combining all pairs of records from R and S where n i=1 simai (R.ai , S.ai ) ≥ τ {{Cite book {{Cite book | last = Mumford | last = Mumford | first = David | first = David <record id=”6627383”> <record id=”6627383”> | authorlink = David Mumford | authorlink = David Mumford <author>David Mumford</author> <author>David Mumford</author> | title = The Red Book of Varieties and Schemes | title = The Red Book of Varieties and Schemes <title>The red book of Varieties and <title>The red book of Varieties and | publisher = [[Springer]] | publisher = [[Springer]] Schemes</title> Schemes</title> | location = Berlin | location = Berlin <publisher>Springer</publisher> <publisher>Springer</publisher> | date = 1999 | date = 1999 <year>1999</year> <year>1999</year> | page = 198 | page = 198 <doi>10.1007/b62130</doi> <doi>10.1007/b62130</doi> | doi = 10.1007/b62130 | doi = 10.1007/b62130 </record> </record> | isbn = 354063293X | isbn = 354063293X }} }} Wikipedia Data set CiteSeer Data set Marcus Paradies Entity Matching for Semistructured Data in the Cloud 5 / 19
- 10. ChuQL Marcus Paradies Entity Matching for Semistructured Data in the Cloud 6 / 19
- 11. ChuQL ChuQL by example Wordcount in ChuQL 1 mapreduce { 2 input { fn : collection (" hdfs :// wiki /") } 3 rr { for $rev in $hxml : in // revision 4 return {" key ": fn : data ( $x // username | $x // ip ) , 5 " val ": $x // title } } 6 map { $hxml : in } 7 reduce { {" key ": $hxml : in = >" key " , " value ": fn : count ( $hxml : in = >" val ")} } 8 rw { < author name ="{ $hxml : in = >" key "}" count ="{ $hxml : in = >" val "}"/ > } 9 output { fn : put (" hdfs :// outputdir /") } 10 } Marcus Paradies Entity Matching for Semistructured Data in the Cloud 7 / 19
- 12. Entity Matching Marcus Paradies Entity Matching for Semistructured Data in the Cloud 8 / 19
- 13. Entity Matching What is Entity Matching? Marcus Paradies Entity Matching for Semistructured Data in the Cloud 9 / 19
- 14. Entity Matching What is Entity Matching? Challenges Entity Matching has quadratic runtime behavior Entity Matching has high CPU- and memory demands The definition of “what is similar” is domain-dependent Marcus Paradies Entity Matching for Semistructured Data in the Cloud 9 / 19
- 15. Entity Matching Entity Matching Architecture b11 b Data Data Source Source S11 S b22 b Match Match Blocking Blocking Matching Matching Result Result R R b33 b Data Data ... Source Source S22 S bnn b Marcus Paradies Entity Matching for Semistructured Data in the Cloud 10 / 19
- 16. Entity Matching Entity Matching Architecture b11 b Data Data Source Source S11 S b22 b Match Match Blocking Blocking Matching Matching Result Result R R b33 b Data Data ... Source Source S22 S bnn b How can we improve the runtime of an EM task? Marcus Paradies Entity Matching for Semistructured Data in the Cloud 10 / 19
- 17. Entity Matching Entity Matching Architecture b11 b Data Data Source Source S11 S b22 b Match Match Blocking Blocking Matching Matching Result Result R R b33 b Data Data ... Source Source S22 S bnn b Distributed Blocking Marcus Paradies Entity Matching for Semistructured Data in the Cloud 10 / 19
- 18. Entity Matching Entity Matching Architecture b11 b Data Data Source Source S11 S b22 b Match Match Blocking Blocking Matching Matching Result Result R R b33 b Data Data ... Source Source S22 S bnn b Distributed Blocking Parallel Matching Marcus Paradies Entity Matching for Semistructured Data in the Cloud 10 / 19
- 19. MAXIM: Entity Matching in the Cloud Marcus Paradies Entity Matching for Semistructured Data in the Cloud 11 / 19
- 20. MAXIM: Entity Matching in the Cloud Requirements and Approach Requirements Efficient processing of semistructured data Scalability to large datasets Independency from specific similarity functions Ability to easily add new similarity functions Marcus Paradies Entity Matching for Semistructured Data in the Cloud 12 / 19
- 21. MAXIM: Entity Matching in the Cloud Requirements and Approach Requirements Efficient processing of semistructured data Scalability to large datasets Independency from specific similarity functions Ability to easily add new similarity functions Main Idea Use MapReduce and ChuQL to process semistructured data Use a search-based blocking to generate candidate pairs Apply similarity functions to candidate pairs within a block Marcus Paradies Entity Matching for Semistructured Data in the Cloud 12 / 19
- 22. MAXIM: Entity Matching in the Cloud Architecture Search Node 1 Search Node 2 Search Node N Engine Engine Engine Data Node Data Node Data Node ... Hadoop Hadoop Hadoop Full-text Task Tracker Full-text Task Tracker Full-text Task Tracker Index Index Index ChuQL Engine ChuQL Engine ChuQL Engine HDFS HDFS Architecture Hadoop cluster with up to 40 nodes Each node runs a search engine and an attached full-text index Each node runs an in-memory XQuery processor Semistructured data is partitioned and placed on HDFS Marcus Paradies Entity Matching for Semistructured Data in the Cloud 13 / 19
- 23. MAXIM: Entity Matching in the Cloud Processing Stages Search Engines Search Engines HDFS HDFS Three Stages Preparation Stage Blocking Stage Matching Stage Marcus Paradies Entity Matching for Semistructured Data in the Cloud 14 / 19
- 24. MAXIM: Entity Matching in the Cloud Processing Stages Search Engines Search Engines HDFS HDFS Transform Extract Store into full-text Build references references index XML index Extract Wikipedia Extract Wikipedia Index CiteSeerX Index CiteSeerX references references records records Preparation Stage Stage 1: Preparation Stage Extracts references from Wikipedia Reads and transforms records from CiteSeerX Sends CiteSeerX data to local full-text index Marcus Paradies Entity Matching for Semistructured Data in the Cloud 14 / 19
- 25. MAXIM: Entity Matching in the Cloud Processing Stages Search Engines Search Engines HDFS HDFS Transform Extract Store into full-text Build Retrieve Generate references references Get query Store index XML index references query response blocks Extract Wikipedia Extract Wikipedia Index CiteSeerX Index CiteSeerX Generate Semantic Generate Semantic references references records records Block Block Preparation Stage Blocking Stage Stage 2: Blocking Stage Reads extracted references from HDFS Probes full-text index to retrieve candidate publications Assign candidate publications to block(s) Marcus Paradies Entity Matching for Semistructured Data in the Cloud 14 / 19
- 26. MAXIM: Entity Matching in the Cloud Processing Stages Search Engines Search Engines HDFS HDFS Transform Store Extract Store into full-text Build Retrieve Generate Verify references references Get query Store record index XML index references query candidate response blocks pairs pairs Extract Wikipedia Extract Wikipedia Index CiteSeerX Index CiteSeerX Generate Semantic Generate Semantic Record pair generation Record pair generation references references records records Block Block Preparation Stage Blocking Stage Matching Stage Stage 3: Matching Stage Read blocks from HDFS Generate candidate pairs and apply similarity functions Store matching pairs and their similarity Marcus Paradies Entity Matching for Semistructured Data in the Cloud 14 / 19
- 27. MAXIM: Entity Matching in the Cloud Stage 1: Preparation Stage Extracting References Indexing Publications Marcus Paradies Entity Matching for Semistructured Data in the Cloud 15 / 19
- 28. MAXIM: Entity Matching in the Cloud Stage 1: Preparation Stage Extracting References Indexing Publications Extraction {{cite journal | author1 = Hansjörg Zeller | author2 = Jim Gray | title = An Adaptive Hash Join Algorithm for Multi-User Environments | journal = Proceedings of the 16th VLDB conference | year = 1990 | pages = 186–197 }} Marcus Paradies Entity Matching for Semistructured Data in the Cloud 15 / 19
- 29. MAXIM: Entity Matching in the Cloud Stage 1: Preparation Stage Extracting References Indexing Publications Extraction {{cite journal | author1 = Hansjörg Zeller | author2 = Jim Gray | title = An Adaptive Hash Join Algorithm for Multi-User Environments | journal = Proceedings of the 16th VLDB conference | year = 1990 | pages = 186–197 }} Transformation <reference type=“journal“> <author1>Hansjörg Zeller</author1> <author2>Jim Gray</author2> <title>An Adaptive Hash Join Algorithm for Multi-User Environments</title> <journal>Proceedings of the 16th VLDB conference</journal> <year>1990</year> <pages>186–197</pages> </reference> Marcus Paradies Entity Matching for Semistructured Data in the Cloud 15 / 19
- 30. MAXIM: Entity Matching in the Cloud Stage 1: Preparation Stage Extracting References Indexing Publications HDFS Extraction {{cite journal | author1 = Hansjörg Zeller | author2 = Jim Gray | title = An Adaptive Hash Join Algorithm for Multi-User Environments | journal = Proceedings of the 16th VLDB conference | year = 1990 | pages = 186–197 }} Transformation <reference type=“journal“> <author1>Hansjörg Zeller</author1> <author2>Jim Gray</author2> <title>An Adaptive Hash Join Algorithm for Multi-User Environments</title> <journal>Proceedings of the 16th VLDB conference</journal> <year>1990</year> <pages>186–197</pages> </reference> Marcus Paradies Entity Matching for Semistructured Data in the Cloud 15 / 19
- 31. MAXIM: Entity Matching in the Cloud Stage 1: Preparation Stage Extracting References Indexing Publications HDFS Extraction Read and Transformation {{cite journal | author1 = Hansjörg Zeller | author2 = Jim Gray <doc> | title = An Adaptive Hash Join Algorithm for Multi-User Environments <field name="id">10.1.1.49.2550</field> | journal = Proceedings of the 16th VLDB conference <field name="title">Selecting Tense, Aspect, and | year = 1990 Connecting Words In Language Generation</field> | pages = 186–197 <field name="author">Bonnie Dorr</field> }} <field name="description">Generating language ...</field> </doc> Transformation <reference type=“journal“> <author1>Hansjörg Zeller</author1> <author2>Jim Gray</author2> <title>An Adaptive Hash Join Algorithm for Multi-User Environments</title> <journal>Proceedings of the 16th VLDB conference</journal> <year>1990</year> <pages>186–197</pages> </reference> Marcus Paradies Entity Matching for Semistructured Data in the Cloud 15 / 19
- 32. MAXIM: Entity Matching in the Cloud Stage 1: Preparation Stage Extracting References Indexing Publications HDFS Extraction Read and Transformation {{cite journal | author1 = Hansjörg Zeller | author2 = Jim Gray <doc> | title = An Adaptive Hash Join Algorithm for Multi-User Environments <field name="id">10.1.1.49.2550</field> | journal = Proceedings of the 16th VLDB conference <field name="title">Selecting Tense, Aspect, and | year = 1990 Connecting Words In Language Generation</field> | pages = 186–197 <field name="author">Bonnie Dorr</field> }} <field name="description">Generating language ...</field> </doc> Transformation Indexing <reference type=“journal“> <author1>Hansjörg Zeller</author1> <author2>Jim Gray</author2> <title>An Adaptive Hash Join Algorithm for Multi-User Environments</title> Lucene Lucene <journal>Proceedings of the 16th VLDB conference</journal> Index Index <year>1990</year> <pages>186–197</pages> </reference> Marcus Paradies Entity Matching for Semistructured Data in the Cloud 15 / 19
- 33. MAXIM: Entity Matching in the Cloud Stage 2: Blocking Stage Block generation Each reference generates a set of candidate publications Each candidate publication is inserted into all blocks, which are listed in reference Marcus Paradies Entity Matching for Semistructured Data in the Cloud 16 / 19
- 34. MAXIM: Entity Matching in the Cloud Stage 2: Blocking Stage Block generation Each reference generates a set of candidate publications Each candidate publication is inserted into all blocks, which are listed in reference Example Hashing <citation> <citation> <id>26334893</id> <id>26334893</id> <citation> <cat>Search engine optimization</cat> <cat>Search engine optimization</cat> <id>26334893</id> 10.0.1.1.124 <cat>Hashing</cat> search algorithms</cat> <cat>Internet <cat>Internet search algorithms</cat> Search Engine <cat>Link analysis</cat> <cat>Link analysis</cat> <cat>Join algorithms</cat> <ref> <ref> <ref> 10.0.1.11.23 <type>journal</type> <type>journal</type> <type>journal</type> <author>Taher Haveliwala</author> <author>Taher Haveliwala</author> <author>Hansjörg Zeller</author> send result <author>Jim Gray</author> <year>2003</year> <year>2003</year> Full-Text <year>1990</year> send query Index <pages>56-70</pages> <pages>56-70</pages> <pages>186-197</pages> Eigenvalue <title>The Second <title>The Second Eigenvalue send result <title>An AdaptiveGoogle Matrix</title> ofof the Hash JoinMatrix</title> the Google Algorithm Join for Multiuser Environments</title> <journal>Stanford University <journal>Stanford University <journal>Proceedings of the 16th VLDB algorithms Technical Report</journal> Technical Report</journal> conference</journal> </ref> </ref></ref> 10.0.1.1.124 </citation> </citation> </citation> 10.0.7.23.14 Marcus Paradies Entity Matching for Semistructured Data in the Cloud 16 / 19
- 35. MAXIM: Entity Matching in the Cloud Stage 2: Blocking Stage Distributed Search in MAXIM (a) Send HTTP request (query) Search Node 1 (c) Engine (b) HTTP response (partial result) Data Node Hadoop (c) Collect partial results Full-text Task Tracker Index ChuQL Engine (a) ) (a (a) (a) (b) (b) (b) (b) Search Node 2 Search Node 3 Search Node 4 Search Node 5 Engine Engine Engine Engine Data Node Data Node Data Node Data Node Hadoop Hadoop Hadoop Hadoop Full-text Task Tracker Full-text Task Tracker Full-text Task Tracker Full-text Task Tracker Index Index Index Index ChuQL Engine ChuQL Engine ChuQL Engine ChuQL Engine Marcus Paradies Entity Matching for Semistructured Data in the Cloud 16 / 19
- 36. MAXIM: Entity Matching in the Cloud Stage 3: Matching Stage Applies user-defined similarity functions to candidate pairs Each attribute can be evaluated by a specific similarity function Marcus Paradies Entity Matching for Semistructured Data in the Cloud 17 / 19
- 37. MAXIM: Entity Matching in the Cloud Stage 3: Matching Stage Applies user-defined similarity functions to candidate pairs Each attribute can be evaluated by a specific similarity function Number of candidate pairs n CP = Ci ∗ Ri (1) i=1 n - # of blocks in B1 , . . . , Bn Ri - # of references in block Bi Ci - # of candidate publications in block Bi CP - # of candidate pairs to verify Marcus Paradies Entity Matching for Semistructured Data in the Cloud 17 / 19
- 38. Summary Summary Wikipedia provides many opportunities for research Need for efficiently processing semistructured data is increasing Entity Matching is critical for data integration and data cleaning Entity Matching is difficult to parallelize due to unbalanced data partitions MAXIM parallelizes EM by building blocks of similar records in a classification fashion MAXIM allows to define own similarity functions and computation functions without changing the algorithm Marcus Paradies Entity Matching for Semistructured Data in the Cloud 18 / 19
- 39. “Everything that can be invented has been invented.” (Charles H. Duell, Commissioner, U.S. Office of Patents, 1899) Marcus Paradies Entity Matching for Semistructured Data in the Cloud 19 / 19
- 40. Experiments Scaleup and Speedup 9 2 Ideal Ideal INDEXING-2000 1.8 EXTRACTING-2000 8 EXTRACTING-2000 INDEXING-2000 Speedup = Base Time / New Time Scaleup = Base Time / New Time BLOCKING 1.6 7 MATCHING 1.4 6 1.2 5 1 0.8 4 0.6 3 0.4 2 0.2 1 0 5 10 20 40 5 10 20 40 Number of nodes Number of nodes (a) Speedup for all stages (b) Scaleup for preparation stage Marcus Paradies Entity Matching for Semistructured Data in the Cloud 20 / 23
- 41. Experiments Query Performance 900 RESULTCOUNT-50 Avg. Query Response Time (ms) 800 RESULTCOUNT-100 RESULTCOUNT-150 700 RESULTCOUNT-200 600 500 400 300 200 100 0 5 10 20 40 Number of Nodes Figure: Query Performance for different result set sizes and cluster sizes. Marcus Paradies Entity Matching for Semistructured Data in the Cloud 21 / 23
- 42. Experiments Blocking Accuracy 1.2 Ideal WRONG-ORDER 1.1 MISPLACED-END MISPLACED-ANY MISSING 1 Accuracy 0.9 0.8 0.7 0.6 0.5 0 0.25 0.5 0.75 1.0 Variance Figure: Blocking accuracy for different typographical error classes. Marcus Paradies Entity Matching for Semistructured Data in the Cloud 22 / 23
- 43. Experiments Number of Candidate Pairs 5.5e+006 RSCOUNT-50 5e+006 RSCOUNT-100 RSCOUNT-150 4.5e+006 RSCOUNT-200 Number of candidate pairs 4e+006 3.5e+006 3e+006 2.5e+006 2e+006 1.5e+006 1e+006 500000 0 0.0 0.1 0.25 0.5 0.75 1.0 Variance Figure: Number of candidate pair verifications in the matching stage. Marcus Paradies Entity Matching for Semistructured Data in the Cloud 23 / 23