Are Linked Datasets fit for Open-domain Question Answering? A Quality Assessment
1 like267 views
The document evaluates the quality assessment (QA) of linked open data (LOD) datasets through various performance measures like precision, recall, and overall F1 scores. It examines components of QA systems including search and retrieval, preprocessing, query formulation, and knowledge base characteristics. Additionally, it provides metrics related to data availability, semantic accuracy, data diversity, and trust along with assessments of data slices from DBpedia and Wikidata.
Are Linked Datasets fit for Open-domain Question Answering? A Quality Assessment
- 13. QA systems Quality assessment of the LOD datasets The answer lies here!
- 14. • •
- 15. Digging into the QA system Typical IR system performances measures ● Overall Performance ○ F1 ○ Precision ○ Recall
- 16. Digging into the QA system Data & Component/Module oriented measures ● Search & retrieval module ○ Indexer ○ Retriever ● Preprocessing / Linguistic ○ NLP - POS tags, NER, etc ○ Entity linking & annotation - semantics ○ Relation extraction & annotation ● Query formulation ○ SPARQL conversion ● Datasource/knowledge base ○ Data Typical IR system performances measures ● Overall Performance ○ F1 ○ Precision ○ Recall
- 17. Digging into the QA system Data & Component/Module oriented measures ● Search & retrieval module ○ Indexer ■ Top K words accuracy; P@10, P@1000, etc ○ Retriever ■ Ranking, Re-ranking, MRR, etc ● Preprocessing / Linguistic ○ NLP - POS tags, NER, etc ○ Entity linking & annotation - semantics ○ Relation extraction & annotation ■ annotation accuracy/precision ■ consistency, interlinking, etc ● Query formulation ○ SPARQL conversion ■ conversion accuracy/precision ● Datasource/knowledge base ○ Completeness ○ Data diversity ○ Trust and Provenance ○ Coverage ○ Timeliness (up to date) ○ etc Typical IR system performances measures ● Overall Performance ○ F1 ○ Precision ○ Recall
- 18. Digging into the QA system Data & Component/Module oriented measures ● Search & retrieval module ○ Indexer ■ Top K words accuracy; P@10, P@1000, etc ○ Retriever ■ Ranking, Re-ranking, MRR, etc ● Preprocessing / Linguistic ○ NLP - POS tags, NER, etc ○ Entity linking & annotation - semantics ○ Relation extraction & annotation ■ annotation accuracy/precision ■ consistency, interlinking, etc ● Query formulation ○ SPARQL conversion ■ conversion accuracy/precision ● Datasource/Knowledge base ○ Completeness ○ Data diversity ○ Trust and Provenance ○ Coverage ○ Timeliness (up to date) ○ etc Typical IR system performances measures ● Overall Performance ○ F1 ○ Precision ○ Recall
- 19. • •
- 20. Evaluated in this study
- 25. •
- 30. ● ○ ○ ○ ●
- 31. DBpedia data slice sizes (in MB)Wikidata data slice sizes (in MB)
- 32. Dimension Metric DB_Rest DB_Poli DB_Film DB_Soc Availability EstimatedDereferenceabilityMetric 0.013 0.013 0.012 0.012 EstimatedDereferenceabilityForwardLinksMetric 0.027 0.027 0.027 0.027 NoMisreportedContentTypesMetric 0 1 1 1 RDFAvailabilityMetric 0 0 0 0 EndPointAvailabilityMetric 0 0 0 0 Interlinking EstimatedInterlinkDetectionMetric - - - - EstimatedLinkExternalDataProviders - - - - EstimatedDereferenceBackLinks 0.012 0.014 0.015 0.022 Semantic accuracy OntologyHijacking 1 1 1 1 MisusedOwlDatatypeOrObjectProperties 1 1 1 1 Data diversity HumanReadableLabelling 0.953 0.985 0.997 1 MultipleLanguageUsageMteric 1 2 3 3 Trust and Provenance Basic Provenance 0 0 0 0 Extended Provenance 0 0 0 0 Provenance Richness 0 0 0 0 DBPEDIA SLICE ASSESSMENT RESULTS
- 33. WIKIDATA SLICE ASSESSMENT RESULTS Dimension Metric Wiki_Rest Wiki_Poli Wiki_Film Wiki_Soc Availability EstimatedDereferenceabilityMetric 0.051 0.063 0.048 0.062 EstimatedDereferenceabilityForwardLinksMetric 0.093 0.053 0.050 0.064 NoMisreportedContentTypesMetric 0 1 0 1 RDFAvailabilityMetric 0 0 0 0 EndPointAvailabilityMetric 0 0 0 0 Interlinking EstimatedInterlinkDetectionMetric - - - - EstimatedLinkExternalDataProviders 5 11 9 8 EstimatedDereferenceBackLinks 0.013 0.098 0.089 0.083 Semantic accuracy OntologyHijacking 1 1 1 1 MisusedOwlDatatypeOrObjectProperties 1 1 1 1 Data diversity HumanReadableLabelling 0.175 0.076 0.091 0.102 MultipleLanguageUsageMteric 2 3 2 3 Trust and Provenance Basic Provenance 0 0 0 0 Extended Provenance 0 0 0 0 Provenance Richness 0.055 0.083 0.010 0.025