Perplexity of Index Models over Evolving Linked Data
Download as PPTX, PDF1 like746 views
This document presents an evaluation of index models for evolving linked open data (LOD), focusing on their stability and accuracy. It describes the structure and metrics used for assessing the models, including Jaccard similarity and perplexity. The findings indicate generally good stability in schema-level indices and suggest future work on quality assessment and query accuracy.
Perplexity of Index Models over Evolving Linked Data
- 1. Institute for Web Science & Technologies – WeST Perplexity of Index Models over Evolving Linked Data Thomas Gottron, Christian Gottron May 27th, 2014 ESWC, Crete
- 2. Thomas Gottron ESWC 27.5.2014, 2Perplexity of Index Models Over Evolving LOD Motivation Index Once upon a time... ... some time later New index ??? Accuracy?
- 3. Thomas Gottron ESWC 27.5.2014, 3Perplexity of Index Models Over Evolving LOD Index Models Over Linked Data
- 4. Thomas Gottron ESWC 27.5.2014, 4Perplexity of Index Models Over Evolving LOD Data Format Linked Data as N-Quads: triple – what is the information? context URI – where does it come from? s op c ( )s op c
- 5. Thomas Gottron ESWC 27.5.2014, 5Perplexity of Index Models Over Evolving LOD Ã( ) (Abstract) Index Models D : Data elements to be retrieved (payload) K : Key elements to access the data (index elements) σ : Selection function: How to get data for a key k1 k2 k3 ... kn d1,1 d1,2 d1,3 ... d2,1 d2,2 d3,1 d3,2 d3,3 ... dn,1 dn,2 dn,3 ... DK s Searchdata structure Efficientstorage andretrieval Data items (payload)Keys
- 6. Thomas Gottron ESWC 27.5.2014, 6Perplexity of Index Models Over Evolving LOD Concrete Example: Subject Based Index Model ukob:Gottron ukob:Staab ukob:Schegi ... tud:CGottron (ukob:Gottron, rdf:type, foaf:Person) (ukob:Gottron, foaf:knows, ukob:Staab) ... (ukob:Staab, swrc:institution, ukob:WeST) (ukob:Staab, foaf:name, „Steffen Staab“) ... (ukob:Schegi, rdf:type, foaf:Person) (ukob:Schegi, foaf:name, „Stefan Scheglmann“) (tud:CGottron, swrc:institution, tud:KOM) (tud:CGottron, foaf:knows, ukob:Gottron) ...
- 7. Thomas Gottron ESWC 27.5.2014, 7Perplexity of Index Models Over Evolving LOD 12 Implemented Index Models Triple based Meta data Schema-level https://github.com/gottron/lod-index-models s ops s opp s opo s opterm s opc s opPLD type s SchemEX s t t st t p p sp p p-1 p-1 op-1p-1 t p sp t
- 8. Thomas Gottron ESWC 27.5.2014, 8Perplexity of Index Models Over Evolving LOD Index Accuracy over Evolving Data
- 9. Thomas Gottron ESWC 27.5.2014, 9Perplexity of Index Models Over Evolving LOD Comparing Indices Once upon a time... ... some time later ???k1 k2 k3 ... kn d1,1 d1,2 d1,3 ... d2,1 d2,2 d3,1 d3,2 d3,3 ... dn,1 dn,2 dn,3 k1 k2 k3 ... kn d1,1 d1,2 d1,3 ... d2,1 d2,2 d3,1 d3,2 d3,3 ... dn,1 dn,2 dn,3
- 10. Thomas Gottron ESWC 27.5.2014, 10Perplexity of Index Models Over Evolving LOD Metrics First indicator of interest: Stability of the key element set Jaccard K1,K2( )= K1 ÇK2 K1 ÈK2 Relative size of the overlap of two sets Jaccard Similarity
- 11. Thomas Gottron ESWC 27.5.2014, 11Perplexity of Index Models Over Evolving LOD How to Measure Accuracy? Queries? No established query log for used data set Different key elements require different queries Cover all of the index Distributions! Relevant to several applications Established metrics for comparison
- 12. Thomas Gottron ESWC 27.5.2014, 12Perplexity of Index Models Over Evolving LOD Obtaining a Distribution from an Index k1 k2 k3 ... kn d1,1 d1,2 d1,3 ... d2,1 d2,2 d3,1 d3,2 d3,3 ... dn,1 dn,2 dn,3 ... Ã D( )K s
- 13. Thomas Gottron ESWC 27.5.2014, 13Perplexity of Index Models Over Evolving LOD Obtaining a Distribution from an Index k1 k2 k3 ... kn 4 2 10 8 K s(k) count Relative frequencies ... K p P k( )= s(k) M M
- 14. Thomas Gottron ESWC 27.5.2014, 14Perplexity of Index Models Over Evolving LOD Comparing Indices Once upon a time... ... some time later ??? K q K p
- 15. Thomas Gottron ESWC 27.5.2014, 15Perplexity of Index Models Over Evolving LOD Comparing Distributions Information theoretic measures ??? KK q p H P( )= - P(x)ld(P(x)) xÎK å Entropy of P Expected length (in bits) for an optimal encoding of a (randomly chosen) key
- 16. Thomas Gottron ESWC 27.5.2014, 16Perplexity of Index Models Over Evolving LOD Metrics H P,Q( )= - P(x)ld(Q(x)) xÎK å Expected length when the encoding is based on a different distribution Cross-Entropy of P and Q PP P,Q( )= 2H(P,Q) How many uniformly distributed keys would have the same entropy Perplexity PP P,Q( )Norm = 2H (P,Q) K Perplexity relative to a uniform distribution over the keys Normalized Perplexity
- 17. Thomas Gottron ESWC 27.5.2014, 17Perplexity of Index Models Over Evolving LOD Metrics: How to Interpret Perplexity Perplexity based on cross entropy „How surprised are you about the outcome of an experiment given you have some expections?“ p 1 10 1 2 1 2 3 4 5 6 PP = 22.585 = 6 q 1 6 1 2 3 4 5 6 PPNorm = 22.585 6 =1 Unfair die model
- 18. Thomas Gottron ESWC 27.5.2014, 18Perplexity of Index Models Over Evolving LOD q 1 10 1 2 1 2 3 4 5 6 Metrics: How to Interpret Perplexity Perplexity based on cross entropy „How surprised are you about the outcome of an experiment given you have some expections?“ PP = 22.161 = 4.472 PPNorm = 0.745 p 1 10 1 2 1 2 3 4 5 6 Unfair die
- 19. Thomas Gottron ESWC 27.5.2014, 19Perplexity of Index Models Over Evolving LOD Metrics: How to Interpret Perplexity Perplexity based on cross entropy „How surprised are you about the outcome of an experiment given you have some expections?“ PP = 22.287 = 4.880 q 3 25 2 5 1 2 3 4 5 6 PPNorm = 0.813 p 1 10 1 2 1 2 3 4 5 6 Unfair die
- 20. Thomas Gottron ESWC 27.5.2014, 20Perplexity of Index Models Over Evolving LOD Metrics: How to Interpret Perplexity Perplexity based on cross entropy „How surprised are you about the outcome of an experiment given you have some expections?“ PP = 23.090 =8.513 q 1 2 3 4 5 6 1 10 1 2 PPNorm =1.418 p 1 10 1 2 1 2 3 4 5 6 Unfair die
- 21. Thomas Gottron ESWC 27.5.2014, 21Perplexity of Index Models Over Evolving LOD Stability of Index Models over Evolving Data
- 22. Thomas Gottron ESWC 27.5.2014, 22Perplexity of Index Models Over Evolving LOD Comparing Indices Once upon a time... ... some time later Jaccard Perplexity k1 k2 k3 ... kn d1,1 d1,2 d1,3 ... d2,1 d2,2 d3,1 d3,2 d3,3 ... dn,1 dn,2 dn,3 k1 k2 k3 ... kn d1,1 d1,2 d1,3 ... d2,1 d2,2 d3,1 d3,2 d3,3 ... dn,1 dn,2 dn,3
- 23. Thomas Gottron ESWC 27.5.2014, 23Perplexity of Index Models Over Evolving LOD Experimental Setup Index construction / Estimation of distributions ... T0 (Base) ... ... T1 T2 T3 TnTn-1 T0 „deviation“ T1 T2 T3 TnTn-1
- 24. Thomas Gottron ESWC 27.5.2014, 24Perplexity of Index Models Over Evolving LOD Results: Jaccard Similarity of Key Set
- 25. Thomas Gottron ESWC 27.5.2014, 25Perplexity of Index Models Over Evolving LOD Results: Normalised Perplexity
- 26. Thomas Gottron ESWC 27.5.2014, 26Perplexity of Index Models Over Evolving LOD Results: Normalised Perplexity (Zoom in)
- 27. Thomas Gottron ESWC 27.5.2014, 27Perplexity of Index Models Over Evolving LOD Conclusion Summary Evaluation of stability of 12 LOD index models Application independent evaluation framework Good stability of schema-level indices Future Work Index specific assessment of quality based on samples Accuracy in answering queries
- 28. Thomas Gottron ESWC 27.5.2014, 28Perplexity of Index Models Over Evolving LOD Thanks! Contact: Thomas Gottron WeST – Institute for Web Science and Technologies Universität Koblenz-Landau gottron@uni-koblenz.de #eswc2014GottronTC