Combining a co-occurrence-based and a semantic measure for entity linking
- 1. Motivation Data on the Web 29/05/13ESWC 2013, Montpellier, France Some eyecatching opener illustrating growth and or diversity of web data Combining a co-occurrence-based and a semantic measure for entity linking ESWC 2013: Extended Semantic Web Conference 28 May 2013, Montpellier, France Bernardo Pereira Nunes, Stefan Dietze, Marco Antonio Casanova, Ricardo Kawase, Besnik Fetahu, Wolfgang Nejdl (PUC-Rio, BR) (L3S Research Center, DE)
- 2. Outline – Introduction – Motivation Example – A combined approach towards entity linking • Semantic Connectivity Score – Katz Index • Co-occurrence-based measures • Combined entity linking approach – Evaluation – Results – Conclusions 29/05/13 ESWC 2013 – Montpellier, France
- 3. Introduction • Linked Data and Web resources • Sparsely interlinked resources • Knowledge bases, with structured knowledge about entities • NER & NED for extraction of entities • Few semantics relationships between entities (skos:related, so:related) • Entity linking, meaningful only at first (direct) degree of connectivity • Exhaustive process considering large amounts of resources 29/05/13 ESWC 2013 – Montpellier, France
- 4. Motivation Example • Semantic relatedness of concepts (entities) • Exploit existing knowledge base structures • Resource semantic similarity (entities) • Latent relationships via semantic relations 29/05/13 ESWC 2013 – Montpellier, France • The Charlotte Bobcats could go from the NBA’s worst team to its best bargain. •The New York Knicks got the big-game performances they desperately needed from Carmelo Anthony and Amar’e Stoudemire to beat the Miami Heat.
- 5. A combined entity-linking approach Novel approach on entity-linking for resources of same and disparate datasets. 1. Semantic Connectivity Score (SCS)– knowledge graph based on Social Network Theory – Katz Index. 2. Co-occurrence based Measure (CBM) – utilise entity co-occurrence in the Web. 29/05/13 ESWC 2013 – Montpellier, France
- 6. Semantic Connectivity Score - SCS • Measure relatedness of entity pairs computing Katz’s Index • Use transversal properties to compute relatedness • Exclude hierarchical properties: – rdfs:subClassOf – dcterms:subject – skos:broader • Quantify semantic connectivity of entity pairs (e1, e2): 29/05/13 ESWC 2013 – Montpellier, France ∑= >< ⋅= τ β 1 ),(21 ||),( 21 l l ee l pathseeSCS transversal paths of length l between entity pairsdamping factor, exponentially penalize longer paths.
- 7. Semantic Connectivity Score – SCS (1) 29/05/13 ESWC 2013 – Montpellier, France • Remove edge directions from graphs • Inverse properties considered equivalent: i.e. isFathorOf ↔ isSonOf • Empirically determine path length Adoptions to knowledge graphs towards applying Katz index measure Inverse property equivalence
- 8. Semantic Connectivity Score – SCS (2) • Optimization factors for Katz: – Exponentially many paths, measuring entity pair relatedness – Small world assumptions – Tradeoff of path length and connectivity contribution (τ=4) 29/05/13 ESWC 2013 – Montpellier, France #Paths with increasing length Computation time for increasing path length
- 9. Co-occurrence-based Measure (CBM) • Approximate number of Web resources mentioning entity pairs • Similar to Pointwise Mutual Information and Normalised Google Distance • Query search engines: e.g. “Carmelo Anthony” + “Charlotte Bobcats” • Extract occurrences of each entity, and as well the entity pairs 29/05/13 ESWC 2013 – Montpellier, France ⋅ === =∨= = otherwise, ))(log( )),(log( ))(log( )),(log( 1),()()(if,1 0)(0)(if,0 ),( 2 21 1 21 2121 21 21 ecount eecount ecount eecount eecountecountecount ecountecount eeCBM
- 10. A combined entity-linking approach • SCS as an exhaustive entity-linking procedure • CBM –search engines to measure relatedness based on entity co-occurrence • Complementary entity-linking results • A combined measure, scalable and with broader coverage: 29/05/13 ESWC 2013 – Montpellier, France > =+ otherwise),,( 0),(if),,( ),( ji jiji jiSCSCBM eeSCS eeCBMeeCBM eeα
- 11. Evaluation Setup • Dataset: USAToday news – 40, 000 document and 80, 000 entity pairs • Gold standard generated using human evaluators – 600 document and 1000 entity assessed pairs • Quantify connectivity with 5-point Likert scale: – correctness: strongly disagree to strongly agree – expectedness: extremely unexpected to extremely expected • Compare CBM, SCS, ESA entity-linking approaches • Standard performance metrics: precision/recall/F1 measure 29/05/13 ESWC 2013 – Montpellier, France
- 12. Entity-Linking Results • 5-point Likert scale, entity connectivity based on gold standard: 29/05/13 ESWC 2013 – Montpellier, France Strongly Agree Agree Undecided Disagree Strongly Disagree 63 178 127 227 217 Precision/Recall/F1 compared against the gold standard for the competing approaches.
- 13. Entity-linking Results (1) • Analysis of uncovered entity connections from competing approaches • Expectedness of uncovered entity connections: – SCS – 25% unexpected novel entity links – CBM – 16% unexpected novel entity links 29/05/13 ESWC 2013 – Montpellier, France CBM (not in SCS) CBM (not in ESA) SCS (not in CBM) SCS (not in ESA) ESA (not in CBM) ESA (not in SCS) Strongly Agree 9.5% 76% 3.1% 71% 7.9% 9.5% Agree 12.3% 63.4% 11.2% 60.1% 8.9% 6.7% Undecided 9.4% 60.6% 6.3% 59.8% 5.5% 7.9% Disagree 15.0% 63.0% 7.1% 53.3% 7.1% 5.3% Strongly Disagree 18.4% 63.1% 51.6% 4.6% 4.6% 6.9%
- 14. Entity-Linking Result Analysis • Connectivity agreement: SCS vs. CBM • Measured agreement based on Kendall’s correlation coefficient: 29/05/13 ESWC 2013 – Montpellier, France Entity connectivity agreement, from connections induced by CBM and supported by SCS τ k@2 k@5 k@10 USAToday 0.40 0.47 0.52
- 15. Entity-Linking Results Analysis (1) • Complementary entity connections between SCS and CBM • Many entity connections labelled as “undecided”, correct • Examples: “Baracak Obama” and “Olympia Snowe” – Human evaluators, marked as not connected – SCS uncovered a connection of length 2 and more 29/05/13 ESWC 2013 – Montpellier, France CBM SCS ESA CBM+SCS Precision 0.32 0.34 0.16 0.34 Recall (GS) 0.81 0.78 0.23 0.90 Recall 0.52 0.51 0.15 0.58 F1 (GS) 0.46 0.47 0.19 0.50 F1 0.40 0.41 0.15 0.43
- 16. Conclusions • An entity-linking approach across disparate datasets • Knowledge graphs, adapted and utilized to uncover entity connections via SCS and CBM • Balanced tradeoffs between information gain and processing time for SCS • Entity-linking gold standard measuring correctness of connectivity and expectedness • Combination of SCS and CBM as scalable entity-linking approach • Increased precision and recall based on SCS+CBM • Correctly uncovered connections marked as irrelevant by human evaluators 29/05/13 ESWC 2013 – Montpellier, France
- 17. Future Work • Exploit semantics of edges connecting entities • Detailed distinction of edges based on entity types • Gold standard improvement, by showing the trace of intermediary entities helping uncover a connection between an entity pair • Filtering of nodes from a knowledge graph to improve scalability 29/05/13 ESWC 2013 – Montpellier, France
- 18. Thank you! Questions? 29/05/13 ESWC 2013 – Montpellier, France