User-driven Quality Evaluation of DBpedia
0 likes1,134 views
The document presents an evaluation of data quality in DBpedia through a user-driven quality assessment methodology and semi-automatic evaluation. It outlines a four-step process for data quality assessment, highlights quality issues identified, and provides statistics from the evaluations conducted. The conclusion discusses the need for ongoing improvements and future assessments of linked open data sources.
User-driven Quality Evaluation of DBpedia
- 1. User-driven Quality Evaluation of DBpedia Amrapali Zaveri, Dimitris Kontokostas, Mohamed A. Sherif, Lorenz Bühmann, Mohamed Morsey, Sören Auer, Jens Lehmann
- 2. Outline ❏Data Quality ❏Data Quality Assessment Methodology ❏Evaluating Quality of Dbpedia ❏ Manual ❏ Semi-automatic ❏Results ❏Conclusion & Future Work
- 3. Data Quality ● Data Quality (DQ) is defined as: ○ fitness for a certain use case* ● On the Data Web - varying quality of information covering various domains ● High quality datasets ○ curated over decades - e.g. life science domain ○ crowdsourcing process - extracted from unstructured and semi-structured information, e.g. DBpedia * J. Juran. The Quality Control Handbook. McGraw-Hill, New York, 1974.
- 4. Data Quality Assessment Methodology 4 Step Methodology: ❏ Step 1: Resource selection ❏ Per Class ❏ Completely random ❏ Manual ❏ Step 2: Evaluation mode selection ❏ Manual ❏ Semi-automatic ❏ Automatic ❏ Step 3: Resource evaluation ❏ Step 4: DQ improvement ❏ Direct ❏ Indirect
- 5. Evaluating Quality of Dbpedia – Manual ❏Phase 1: Creation of quality problem taxonomy ❏Phase 2: User-driven quality assessment
- 6. Evaluating Quality of Dbpedia – Manual ❏Phase 1: Creation of quality problem taxonomy ❏Phase 2: User-driven quality assessment
- 7. Quality Problem Taxonomy Dimension Category Sub-category D F Dbpedia Specific Accuracy Triple Incorrectly extracted Object value is incompletely extracted - E - Object value in incorrectly extracted - E - Special template not properly recognized √ E √ Datatype problems Datatype incorrectly extracted √ E - Implicit relation- ships between attributes One fact is encoded in several attributes - M √ Several facts are encoded in one attribute - E - Attribute value computed from another attribute value - E + M √ D = Detectable means problem detection can be automized. F = Fixable means the issue is solvable by amending either the extraction framework (E), the mappings wiki (M) or Wikipedia (W).
- 8. Quality Problem Taxonomy Dimension Category Sub-category D F Dbpedia Specific Relevancy Irrelevant inform- ation extracted Extraction of attributes containing layout information √ E √ Redundant attribute values √ - - Image related information √ E √ Other irrelevant information √ E - Represen- tational Consistency Representation of number values Inconsistency in representation of number values √ W - Interlinking External links External websites √ W - Interlinks with other datasets Links to Wikimedia √ E - Links to Freebase √ E - Links to Geospecies √ E - Links generated via Flickr wrapper √ E -
- 9. Evaluating Quality of Dbpedia – Manual ❏Phase 1: Creation of quality problem taxonomy ❏Phase 2: User-driven quality assessment
- 10. User-driven quality assessment Type Contest-based Participants LD experts Task Detect and classify LD quality issues Time 1 month Reward 300 EU prize Tool TripleChekMate Crowdsourcing HITs (Human Intelligent Tasks), Submit to a crowdsourcing platform (e.g. Amazon Mechanical Turk) Financial Reward for each HIT
- 11. DQ Assessment Tool - TripleCheckMate http://nl.dbpedia.org:8080/TripleCheckMate-Demo/
- 12. Evaluating Results - Manual Methodology Total no. of users 58 Total no. of distinct resources evaluated 521 Total no. of resources evaluated 792 Total no. of distinct resources without problems 86 Total no. of distinct resources with problems 435 Total no. of distinct incorrect triples 2928 Total no. of distinct incorrect triples in the dbprop namespace 1745 Total no. of inter-evaluations 268 No. of resources with evaluators having different opinions 89 Resource-based inter-rater agreement (Cohen’s kappa) 0.34 Triple-based inter-rater agreement (Cohen’s kappa) 0.38
- 13. Evaluating Results - Manual Methodology No. of triples evaluated for correctness 700 No. of triples evaluated to be correct 567 No. of triples evaluated incorrectly 133 % of triples correctly evaluated 81 Average no. of problems per resource 5.69 Average no. of problems per resource in the dbprop namespae 3.45 Average no. of triples per resource 47.19 % of triples affected 11.93 % of triples affected in the dbprop namespace 7.11
- 14. Evaluating Quality of Dbpedia – Semi-automatic ❏ Step 1: Automatic creation of an extended schema ❏ DL-Learner* ❏ for all properties in DBpedia, axioms expressing the (inverse) functional, irreflexive and asymmetric characteristic were generated ❏ minimum confidence value of 0.95 ❏ Step 2: Manual evaluation of the generated axioms ❏ 100 random axioms per type ❏ Restricted evaluation of those axioms where at least one violation is found ❏ Taking target context into account *J. Lehmann. DL-Learner: learning concepts in description logics. Journal of Machine Learning Research (JMLR), 10:2639{2642, 2009.
- 15. Evaluation Results - Semi-automatic ❏ Irreflexivity: ❏ dbpedia:2012_Coppa_Italia_Final dbpedia-owl:followingEvent dbpedia:2012_Coppa_Italia_Final ❏ Asymmetry: ❏ dbpedia-owl:starring with domain Work and range Actor ❏ Functionality: ❏ 2 different values 2600.0 and 1630.0 for the density of the moon Himalia. ❏ Inverse Functionality: ❏ Domain: dbpedia-owl:FormulaOneRacer Range:dbpedia-owl:GrandPrix Violation: dbpedia:Fernando_Alonso dbpedia-owl:firstWin dbpedia:2003_Hungarian_Grand_Prix . dbpedia:WikiProject_Formula_one dbpedia-owl:firstWin dbpedia:2003_Hungarian_Grand_Prix .
- 16. Evaluation Results - Semi-automatic methodology Characteristic #Properties Correct #Violation Total Violated Min Max Avg. Total Irreflexivity 142 24 24 1 133 9.8 236 Asymmetry 500 144 81 1 628 16.7 1358 Functionality 739 671 76 1 91581 2624.7 199480 Inverse Functionality 52 49 13 8 18236 1685.2 21908
- 17. Conclusion & Future Work ● Empirical quality analysis for more than 500 resources of a large linked dataset extracted from crowdsourced content ● Future work: ○ Fix problems detected (Improvement step) ○ Assess other LOD sources ○ Adopt an agile methodology to improve quality of LOD ○ Revisit quality analysis (in regular intervals)