Profiling Linked Open Data

Editor's Notes

• #5: According to the definition we found in the Encyclopedia of Database Systems, The process of evaluating data quality is called data profiling and typically involves gathering several aggregated data statistics which constitute the data profiling. So we can say, that data profiling refers to the activity of creating small but informative summeries of a dataset and is a cardinal activity when facing an unfamiliar dataset.
• #6: In the actual LOD State we have 1014 datasets, 188 mio triples, overall 7 categories, 80% of the linking property is owl:sameAs, 98,22% of datasets use RDF vocabulary, around 8% of the datasets provide licensing information etc. Even in this huge amount of data, we still miss information which might be hidden or not easy to understand
• #7: In this amount of data we still make some questions: What types of resources are described in a data set? How are they described? How well connected are the datasets in the LOD cloud? What is their topic/s? Are data described as prescribed by the ontology?
• #8: Why we need profiling? The need to profile a new or unfamiliar dataset arises in many situations, in general to prepare for some subsequent task. Data quality assessment: Probably the most typical use case when we need profiling is when we are preparing for a data cleansing process. Profiling reveals data errors, such as inconsistent formatting, missing values or outliners. Profiling results can also be used to measure and monitor the general quality of a dataset. Query optimization: Basic profiling is performed by most of database systems to support query optimization with statistics about tables and columns. These profiling results can be used to estimate for example the cost of a query plan. -Ontology and Data integration: Often the dataset to be integrated are somewhat unfamiliar and to integration expert wants to explore the dataset first. How large is the db? Are there independencies between tables and among dbs. Ontologies published on the Web, even for datasets in similar domains can have differences. Data profiling techniques can help understanding the overlap between ontologies and help in the process of ontology creation, maintenance and integration. Data analytic: Almost any statistical analysis or data mining run is preceded by profiling step to help the analyst have a hint at of the data at hand -Complex schema discovery: Schema complexity leads to difficulties to understand and access the data. Schema summaries provide users a concise overview of the entire schema despite its complexity. -Topical Classification: Finding features that best represent the topic/s of a given dataset can help not only the topical classification of the dataset but also understanding the semantic of the information found in the data. -Data visualization for summarization. Profiling techniques can support data visualization tools to visualize large multidimensional datasets by displaying only a small and concise summary of the most relevant and important features.
• #9: In the table we show an overview of the actual state of the art tools and techniques. Roomba - is a framework which validate and generate descriptive dataset profiles. One of portal API endpoint is queried, and information about dataset’s title, description, maintainer email, update and creation date are extracted. From the extracted metadata they are able to extract resources associated with that dataset. If the dataset contains many instances only a sample of 10% is used for the validation of the extracted metadata. There is no information about automatization, scalability. There is no interface and only the code is available in github. Also no tutorials are provided. LODStats – is used to gather some comprehensive statistics about rdf data. So in input it takes an RDF file and as an output it generates 32 different statistical criteria on schema and data level. Usually the statistics cover classes/properties usage, out links, in links etc. There exist only a demo on how statistics look like and no information about the availability, scalability and license. ExpLOD - supports exploring summaries of RDF usage and interlinking among datasets. As an input it considers the dataset, the BL (bisimulation label graph which is constructed considering the rdf usage of each node considering also its neighbors) schema and the neighborhoods to consider. The summaries produced can be viewed and explored in an interactive graphical environment and they can also be exported in a variety of formats (including RDF). There are no information about automatization, scalability, license or tutorials available. RDFStats – is used to generate statistics about rdf datasets. the generator creates histograms for each combination of class c, property p (used in the extension of c), and type range t of property values. Because a property value can be a (plain or typed) literal, blank node, or URI, it is necessary, to generate different histograms for each of the occurring types. It is a semi automatic tool, because you have to manually configure it. There is no information about the scalability and
• #10: This table has only aggregated information about profiling tools, but in my research I performed a deeper survey for each tool identifying where is possible each statistical criteria which can be calculated by the tool.
• #11: While data profiling techniques have proposed several solutions as the literature shows, it still represents several challenges: First, the results of data profiling are computationally complex to discover. Discovering key candidates or dependencies usually involves some sorting step for each considered column. Different and new data management architectures and frameworks have emerged including distributed systems, multi-core or main-memory based servers etc. Traditional profiling task can not be applied to Linked Data due to their heterogeneity. Heterogeneity can appear in different forms such as different formats or query languages called syntactic heterogeneity. Linked Open Data can be represented in different formats, stored in different storage architectures also the data encoding schemes may vary. This is referred to as schematic heterogeneity. Datasets published as LOD might use different vocabularies, to describe synonymous terms. Although much effort is done in the past what is missing even in the state of the art is a unified view of data profiling as a filed and a unified framework for its tasks.
• #13: Our intent is to develop automatic approaches and generating new statistics that are not covered by the actual state of the art techniques. While much effort is done in the past as described in the state of the art, the statistics that are generated are limited in some basic statistics such as the number of triples, number of classes/ properties that are used in a dataset, the datatypes or sameAs links used, etc. Datasets hold much more interesting information which might be hidden, but at the same time, this information could be useful for the consumer of the dataset. As data profiling is referred to the activity of providing useful descriptive information, new techniques on how to extract the hidden information should be developed. So applying some data mining techniques to extract useful knowledge from large datasets should be investigated. Different data mining techniques, such as association rule mining, can be used to discover and extract patterns and dependencies in the dataset. These patterns might provide useful information especially to detect errors and inconsistencies in spatial data (consistency quality dimension). Also other algorithms such as, Apriori algorithm , can be used to find sets of items within a dataset that satisfies a user-defined minimum support threshold, to support summarization. Implementation of different approaches for outlier detection, like distance/deviation/depth-based, evolutionary techniques, etc. could provide insight about abnormalities in the underlying data. Also as the number of the datasets published is increasing the need to adapt and optimize profiling techniques to support huge amount of data is also high. A good approach when dealing with large datasets, is to improve the profiling performance running the calculation of statistics and patterns extraction in parallel. MapReduce can help for this profiling challenge. We also plan to adapt some data mining techniques to deal with high dimensionality data, such as Linked Open Data A deep literature study for the actual tools which are used to profile Linked Open Data has been taken. We analyzed existing tools in terms of the goal they are used for, techniques, input, output, approach, automatization information, license etc, with the aim to have a complete view of the existing approaches and techniques for profiling. This deepen study will also help us for the third contribution on creating a general methodology for each of the profiling tasks. Also our intend is to provide the topical information of the dataset while giving our profile.
• #16: As a first step we measured the value of Linked Open Data, profiling the data published as Open Data from the Italian Public Administrations calculating a compliance index considering three quality dimensions for the published data; completeness, accuracy and timeliness. Public administrations are compelled by law number 33 of “Decreto sviluppo”, to publish all their data following a strict schema where the information should be placed under a session of their websites called “Amministrazione Trasparente” which can be found in the PA’s homepage, where you can find information about various topics of administration. Finally, the legislative decree 150 imposes to public administrations to publish documents in open format that correspond to the 1 star of the 5 stars scale proposed by Berners-Lee (2006). The law each PA to have 69 sessions on the webpage.
• #17: We have a benchmark with 50 public administration websites considering 3 factors in distribution (geographical distribution, country wide) types of PAs (regions, county and municipalities) and (their size by number of inhabitants). We then calculated the compliance index considering three criteria: the completeness of the website, the accuracy and the timelines. After we normalized all the values and gave a greater weight to completeness we calculated the compliance index with this formula.
• #18: After analyzing all the results, calculated with the above formula, we profiled Italian PAs in terms of completeness, accuracy and timeliness taking in consideration the criteria of the sampling. All these statistics gave an overview of compliance index for all the PAs in Italy. As we can see from the results the average compliance index for regions is 0.488 where more that half of them have a lower index than the mean. The average index is somehow grater for provinces but still 50% of the provinces have a lower index than the mean. While the compliance index is even lower for municipalities. The highest compliance index is for the region of Veneto with 0.839 while Campania has the lowest score 0.043. If we want to see the compliance degree divided by the type of PA, as showed in the table it seems that regions in the centre of Italy have a higher index than those in the south. Even if we consider provinces the higher compliance have the provinces in the north, while for those in the centre and south they are pretty much the same. If we consider the number of inhabitants it seems that PAs with high number of inhabitants seems to be the ones to comply more to the law. For more details you can check the publication of this work.
• #19: To which extent the topical classification of new LOD datasets can be automated for upcoming versions of the LOD cloud diagram using machine learning techniques and the existing annotations as supervision. Beside creating upcoming versions of the LOD cloud diagram, the automatic topical classification of LOD datasets can be interesting for other purposes as well. Facilitating queries for similar datasets discovery; Agents navigating on the Web of Linked Data should know the topical domain of the datasets that they discover by following links in order to judge weather the datasets might be useful for their use case at hand or not. Speeding up data searches. Knowing the domain of a dataset can not only facilitate queries but also speed up data searches, as now agent know what data are they looking for. Furthermore it is interesting to analyze characteristics of datasets grouped by topical domain, so that trends and best practices that exist in that particular topical domain can be identified. These are the main motivations in building to the best of our knowledge the first automatic approach to classify LOD datasets into topical categories that are used by the LOD cloud diagram.
• #20: For this purpose we used the latest version of LOD cloud crawled in April 2014 containing 1014 different LOD datasets describing around 8 million resources. The datasets were manually classified into one of the following categories; media, government, publications, life sciences, geographic, social networking, user generated content and cross domain. As showed in the table The LOD cloud is dominated by datasets belonging to the category social networking (48%), followed by government (18%) and publications (13%) datasets. The categories media and geographic are only represented by less than 25 datasets within the whole corpus. For each of the datasets, we created the following eight feature sets based on the crawled data: Vocabulary Usage (VOC): As many vocabularies target a specific topical domain, eg bibi bibliographic information, we assume that the vocabularies that are used by a dataset might be helpful indicator for determining the topical category of the dataset. Thus we determine the vocabulary of all terms that are used as predicated or as the object of a type statement within each dataset, Altogether we identified 1439 different vocabularies being used by the datasets. Class URIs (Curi): As a more fine-grained feature, the rdfs: and owl:classes which are used to describe entities within a dataset might provide useful information to determine the topical category of the dataset. Thus we extracted all the classes that are used by at least two different datasets resulting in 914 attributed for this feature set. Property URIs (Puri): Beside the class information of an entity, information about which properties are used to describe the entity can be helpful. For example it might make a difference, is a person is described with foaf:knows statements or if her professional affiliation is provided. To leverage this information, we collected all properties that are used within the crawled data by at least two datasets. This feature set consists of 2333 attributes. Local Class Names (LCN): Different vocabularies might contain synonymous (or at least closely related) terms that share the same local name and only differ in their namespace, eg foaf:Person and dbpedia:person. Creating correspondences between similar classes from different vocabularies reduced the diversity of features but on the other side might increase the number of the attributes which are used by more than one dataset. As we lack correspondences between all the vocabularies, we bypass this, by using only the local names of the type URIs, meaning vocab1: Country and vocab2: Country are mapped to the same attribute. We used a simple regular expression to determine the local class name checking for #, : and / within the type object. By focusing only on the local part of a class name, we increase the number of classes that are used by more than one dataset in comparison to Curi and thus generated 1041 attributes for the LCN feature set. Local Property Names (LPN): Using the same assumption as for the LCN feature set, we also extracted the local name of each property that is used by a dataset. This results in treating vocab1:name and vocab2:name as a single property. We used the same heuristic for the extraction as for the LCN feature set and generated 2493 different local property names which are used by more than one dataset, resulting in an increase of the number of attributes in comparison to the Puri feature set. Text from rdf:label (LAB): beside the vocabulary level features, the names of the described entities might also indicate the topical domain of a dataset. We thus extracted all values of rdfs:label properties, lower-case them and tokenized the values at space characters. We further excluded tokens shorter than three and longer than 25 characters. Afterward we calculated the TF-IDF value for each token while excluding tokens that appeared in less than 10 and more than 200 datasets, in order to reduce the influence of noise. This resulted in a feature set consisting of 1440 attributes. Another feature which might help to assign datasets to topical categories is the top-level domain of the dataset. For instance, government data is often hosted in the gov top leve domain, whereas library data might be found more likely on edu or org top level domains. We restrict ourselves to top-level domains and not public suffixes. In & out degree: In addition to the covabulary-based and textual features, the number of outgoing RDF links to other datasets and incoming rdf links from other dataset could also provide useful information for classifying datasets. This features could give a hint about the density of the linkage of a dataset, as well as the way the dataset is interconnected within the whole LOD cloud ecosystem. We were able to create all features (except LAB) for 1001 datasets. As only 470 datasets provide the rdfs:label, we only uses these datasets for evaluating the utility of the LAB feature set.
• #21: We evaluated the following three classification techniques on our task of assigning topical categories to LOD datasets K-Nearest Neighbor classification models make use of similarity between new cases and known cases to predict the class for the new case. A case is classified by its majority vote of its neighbors with the case being assigned to the class most common among its k nearest neighbors measured by the distance function. In our experiments we used a k equal to 5. J49 Decision Tree: A decision tree is a flowchart-like tree structure which is built top-down from a root node and involves some partitioning steps to divide data into subsets that contain instances with similar values. For our experiments we use the Weka implementation of the C4.5 decision tree. We learn a pruned tree, using a confidence threshold of 0.25 with a minimum number of 2 instances per leaf. Naïve Bayes: As a last classification method, we used Naïve Bayes. NB uses joint probabilities of some evidence to estimate the probability of some event. Although this classifier is based on the assumption that all features are independent, which is violated in many use cases, NB has shown to work well in practice. In order to evaluate the performance of the selected classifiers we use a 10-fold cross-validation and report the average accuracy As the total number of occurrences of vocabularies and terms is heavily influenced by the distribution of entities within the crawl for each dataset, we apply two different normalization strategies to the values of the vocabulary level features VOC; Curi, Puri, LCN and LPN. On the one hand side we created a binary version (bin) where the feature vectors of each feature set consist of 0 and 1 indicating presence and absence of the vocabulary or term. While the second version the relative term occurrence (rto) captures the fraction of the vocabulary or term usage for each dataset. As the number of datasets per category as showed in the table, is not equally distributed within the LOD cloud, which might influence the performance of the classification model, we also explore the effect of balancing the training data. We used three different datasets approaches: (1) no sampling, so we applied our model in the data without any sampling techniques, (2) we down sample the number of datasets used for training until each category is represented by the same number of datasets; this number is equal to the number of datasets within the smallest category; and (3) we up sample the datasets for each category until each category is at least represented by the number of datasets equal to the number of datasets of the largest category. While down sampling reduces the chances to overfit a model into the direction of the larger represented classes, it might also remove valuable information from the training set, as examples are removed and not taken into account for learning the model. The up sampling, ensures that all possible examples are taken into account and no information is lost for training, but by creating the same entity many times can result in emphasizing those particular data points. For example a neighborhood based classifier might look at the 5 nearest neighbors, which than could be one and the same data point, which would result into looking only at the nearest neighbor.
• #22: For our second set of experiments, we combine the available attributed from the different feature sets and train again our classification models using the three described algorithms. As before, we generate a binary and relative term occurrence version of the vocabulary-based feature sets. In addition we created a second set (binary and relative term occurrence) where we omit the attributes from the LAB feature set as we wanted to measure the influence of this particular set of attributes, which is only available for less than half of the datasets. Furthermore we created a combined set of attributes from the previous section. We can observe that when selecting a larger set of attribues our model is able to reach a slightly higher accuracy of 81.62% than using just the attributes for the one feature set 80.59% LCNbin).
• #23: In the following we will have a look at the confusion matrix of the best performing approach Naïve bayes trained on attributed of the NoLabbin feature set using the up sampling. On the left side of the matrix we list the predications by the learned model, while the head names the actual category of the dataset. As observed in the table, there are three kinds of errors which occur more frequently than ten times. The most common confusion occurs for the publication domain, where a larger number of datasets are predicted to belong to the government domain. A reason for this is that government datasets often contain metadata about government statistics which are represented using the same vocabularies and terms (e.g. skos:consept) that are also used in the publication domain. This makes it challenging for a vocabulary based classifier to distinguish those two categories apart. Another example is the classification of the dataset of the Ministry of Culture in Spain., which was manually assigned the publication label whereas our model predict government which turns out to be a borderline case in the gold standard. A similar frequent problem is the prediction of life sciences for datasets in the publications category. This can be observed for the ns.nature publication website which describes the publication in Nature. Those publications are often in the life science field which makes the labeling in the gold standard a borderline case. The third most common confusion occurs between the user generated content and the social networking domain. The problem here is in the shared use of similar vocabularies such as foaf. At the same time, labeling a dataset as either one of the two is often not so simple. The social networking dataset should focus in the presentation of people and their interrelations, while user generated content should have a stronger focus on the content, Datasets form the personal blogs, such as wordpress.com however can convey both aspects. Due to the labeling rule these datasets are labeled as user generated content but our approach frequently classifies them as social networking. While we can observe some true classification errors, many of the mistakes made by our approach actually point at datasets which are difficult to classify and which are rather borderline cases between two categories.
• #24: ABSTAT is a framework which can be used to summarize linked datasets and at the same time to provide statistics about them. The summary consists of Abstract Knowledge Patterns (AKPs) of the form <subjectType, predicate,objectType> which represent the occurrence of triples < sub,pred,obj> in the data, such that subjectType is a minimal type of sub and objectType is a minimal type of obj. The ABSTAT summaries can help users comparing in which of two datasets a concept is described with richer and diverse properties, and also help detecting errors in the data, extracting counterintuitive AKPs, e.g.,<dbo:Band,dbo:genre,dbo:Band> . ABSTAT can also be used to x the domain and range information for properties. Either the domain or the range is unspecified for 585 properties in DBpedia Ontology and AKPs can help us in determining at least one domain and one range for the unspeciefid properties. For example, for the property http://dbpedia.org/ontology/governmentType in DBpedia we do not have information about the domain. With our approach we can derive 7 different AKPs meaning that we can derive 7 domains for this property. ABSTAT can be used to detect missing or datatype diversity, etc
• #25: To evaluate the validity of the proposed approach or the results achieved is very difficult as in the field of LOD profiling there is no Gold Standard, thus is very difficult to compare with others. For this issue, we want to further explore how these new statistics or summarization allow to improve the performance of the actual profiling techniques and tools, e.g. how profiling tasks can improve full-text search etc. To evaluate the validity of the proposed profiling techniques to summarize datasets, as pattern discovery is not trivial, humans will evaluate the validity of the summarization in terms of relatedness and informativeness. We intend to provide to users a list of statistics and ask them which in their opinion is more important to support profiling of Linked Open Data. The evaluation of the performance of profiling tasks is very difficult, which still remains an open issue on which I am currently working.
• #27: To sum up, in this work we tried to investigate to which extent the topical classification of new LOD datasets can be automated using machine learning techniques. Our experiments indicate that vocabulary-level features are good indicator for the topical domain yielding an accuracy of around 82%. The analysis of the limitations of our approach i.e the cases where the automatic classification deviates from the manually labeled one, points to a problem of the categorization approach that is currently used for the LOD cloud. All datasets are labeled with exactly one topical category, although sometimes two or more categories would be equally appropriate. Thus the LOD dataset classification task might be more suitably formulated as a multi-label classification problem. A particular challenge of the classification is the heavy imbalance of the dataset categories, with roughly half of the datasets belonging to the social networking domain. A two stage approach might help, in which a first classifier tries to separate the largest category from the rest, while a second classifier then tries to make a prediction for the remaining classes. When regarding the problem as a multi-label problem, the corresponding approach would be a classifier chains, which make a prediction for one class after the other, taking the prediction of the first classifiers into account as a feature for the remaining classifications. In our experiments, RDF link have not been exploited beyond dataset in and out degree. Some link-based classification techniques may further exploit the content of a dataset. Also adding more features to the model might help in achieving higher accuracy such as linkage information