Classifying Crisis Information Relevancy with Semantics (ESWC 2018)

Editor's Notes

• #2: ‘Classifying Crisis-information Relevancy with Semantics’
• #3: As the topic of the paper suggests, crisis situations are the principal motivation behind the work. People around the world are impacted by crisis and disasters in various forms. And in this era with the ability to share and access information in real time they resort to different online social media forums. Twitter certainly is among the most prominent medium for sharing and accessing real time information.
• #4: To support this idea, I would like to highlight this case from Hurricane Harvey in 2017. What you see highlighted are two very crucial piece of information shared in course of crisis situations. A volunteer driven handle, that collates info on a portal about who needs help and rescue, and who could assist in that geographical area. This is an interaction between two parties that resulted in rescuing 3 elderly ladies. But these critical information are not always easy to find and access as we are also well aware of the challenges that social media projects along side the opportunity that it offers.
• #5: But given those challenges, the opportunities these platforms offer have widely been acknowledged by humanitarian and government agencies.
• #6: The need for having tools and systems to rightly determine what is valuable on social media with respect to its relatedness to crisis situations is highlighted in this minor exercise of performing keyword based search on twitter during Hurricane Harvey. It is evident that not everything that might contain crisis specific based terminology is always a related content. That’s precisely where the problem of ‘what is crisis related’ defined.
• #7: But the problem explored in this work isn’t just about what is crisis related and what is not. They key problem is the diversity in the crisis events. Different crisis situations have different types and levels of impact on human life in form of well being, civic facilities, and what not.
• #8: Which results in a very broad spectrum of data. For instance, if we see here the social media data from situations like e’quakes, wildfires, explosions, road situations , terrorism to name a few. You can see how diverse the information is. For that we do need to come up with ways to be able to filter in as much diverse related content as we can.
• #9: So as a requirement of disaster management, we come down to the following questions. Since we understand that manually sieving through the social stream is nearly impossible so can we have automated ways to identify relevant information and can we learn from crisis situations to identify relevancy in new crisis situations.
• #10: Previous approaches have made attempts at tackling this problem. Some have adopted ML approaches where they go either by supervised classification approach or unsupervised. Supervised Approaches: Often making use of n-grams, linguistic features, and/or statistical features of tweets. Unsupervised Approaches: Keyword processing and clustering. Some of the approaches perform semantic enrichment of the data to create a faceted search on top of the semantic data. But that does not strictly tell you if something is crisis related or not. And may require a new strategy to search each time in new type of crisis events.
• #11: Our hypothesis is that the crisis relatedness is exhibited by combination of various concepts that occur in the user generated content. It might not always be just one key term that establishes the crisis relatedness of a tweet. So, we go beyond the statistical features and incorporate the contextual semantics along with the statistical features. We hypothesize that different crisis situations, while they maybe very discrete in their vocabulary and sense, can relate to each other somewhere at a broader contextual sense.
• #12: A number of statistical features can be extracted: length, number of words, various part of speech such as noun, verbs, pronouns, hashtags, and some can also calculate the readability score that scores a tweet on how easy or complicated it is structured to read. Gunning Fox Index is one of the methods to calculate that.
• #13: Various additional semantics can be considered. Co-occurring words (words that are very common to occur together across large scale corpuses). Word-embeddings are a good example of it. Extracted entities- along with the original text can sharpen the context more. Next, we can refer to the knowledge base (such as DBpedia) to retrieve extra information/properties about that entity. We can also use hierarchical context (from WordNet) to retrieve hypernyms, synonyms to each concept to generalise the context more.
• #14: We require NER tools and Knowledge Bases to extract the semantics that we want. Here are some of the well known tools available.
• #15: In this work we have relied on BabelNet knowledge base and NER API built on top of it. BabelNet is a multilingual semantic network resource which incorporates multiple knowledgebase such as wikipedia and wordnet, that really caters to the requirements here.
• #16: Here is an example of using BabelNet to semantically enrich a tweet by extracting entities and their hypernyms. We annotate the key entities in the text and then look for their corresponding hypernyms and augment them to the overall context.
• #17: The features that we have extracted are as follows. The statistical features are… The semantic features are. We extract hypernyms assuming that diverse concepts can relate when the context is expanded to parent levels of concepts. We refer to the Dbpedia properties to retrieve extra information/properties about that entity, which can link us to broader knowledge about its nature.
• #18: To gain a perspective of what we imply by semantic enrichment across the crisis types let us look at the following real scenario tweets.
• #19: Another example elaborating the same.
• #20: As an end to end process in a nutshell, this is what we do.
• #21: CrisisLex is a very popular data repository which has time and again been referred to for various related research studies. We have used one particular data corpus from Crisis Lex, and that is called CrisisLexT26. This dataset comprises of manually labelled tweets of 26 crisis events that occurred between 2012 and 13. Each of these 26 events have close to 1000 labelled tweets. So, here they have 4 labels. We merge the two to create a binary class system.
• #22: To have a balanced learning of both the classes, we ensure to pass an un-skewed data to the classifier. In each of the 26 events we select the same number of related tweets as the unrelated.
• #23: Here you can see related and unrelated distribution across all the 26 crisis events. As it is obvious, there isn’t an equal distribution across each event though. I would like to highlight each event is basically a broad crisis situation and tweets have been collected during that situation and that we call as event. For instance CWF is Colorado Wildfire, and it contains various tweets collected during that event.
• #24: Here we have categorised events based on their types. And this infographic shows that maximum events in this dataset are in flood/typhoons and next is earthquakes. Next we have same number of crash and shooting/bombing events.
• #25: Now we describe out experiment design. We create 4 feature models to evaluate. Statistical which is our baseline. Next, we create 3 semantic models where we enhance the SF feature model with the semantic features. In 3 models, we create SF + Babel Semantics, SF + Dbpedia semantics, and in last model we combine both the Babel and Dbpedia features. When we add the semantics we concatenate the semantics with the original text of the tweet and the do the tokenisation and create n-grams. So, these are the feature models. Now we design the classification methods. First, as a broad run, we simply merge the data from all the events and perform 20 iterations of 5-fold CV to just see how the 4 feature models perform.
• #26: Now here, we begin designing the cross crisis classification methods. Where we aim to evaluate how the classifier performs in while classifying the data from a new crisis event type. We set up 2 criteria. In Criteria 1, we test a type of crisis event when the type is already seen by the classifier in the training data.
• #27: In Criteria 2, we test a type of crisis event when the type is not seen by the classifier in the training data. For instance, we have a classifier trained on data collected during say flood and earthquake events. And we get to classify something coming from a factory collapse situation like here. Basically these 2 criteria set up here are critical part of our analysis.
• #28: We chose SVM linear kernel. SVM is known for its suitability in text classification problems. On top of it we compared the linear kernel’s performance against other kernels and logistic regression following a 20 iteration of 5-fold CV over entire data. SVM Linear Kernel was found to be more statistically significant, and had a better mean F1 value of 0.8118 and a p-value of < 0.00001.
• #29: We look at the results, so this is when we merge the entire data and do cross validation. So while this is not cross-crisis classification strictly, but it still shows that broadly the semantics perform slightly better than the baseline, improvement ranging from 0.6 to 1.4 %.
• #30: Now in Criteria 1, where the event is completely new but the classifier has seen the type of the test event in training. In this case we, take out the test event from the data, and use rest of the dataset as training data. As seen in an earlier infographic, flood/typhoon and earthquake had good number of events in the overall dataset, we chose to perform our analysis on these crisis types. So, when the classifier has already seen a type of crisis event, the semantics may not always be superior to the statistical features as we can observer from the results. The Dbpedia semantics seem to be the more consistent of the semantic feature model, performing better than the baseline in 6 out of 11 test events.
• #31: Now, when we see the Criteria 2, where the type of the event is not seen in the training data. In this case, while we test an event of a given type we ensure that none of the other events of similar type are in the training data. Here we see that firstly the performance of baseline drops significantly in comparison to baseline in Criteria 1. Secondly the semantic models, particularly Dbpedia feature model, outperforms the baseline in 10 out of 15 test events. As an additional category of types we also included the events from train crash, bombing/terror attacks.
• #32: To analyse how the semantics were effecting the nature of the data and the classifier, we performed Information Gain across all the feature models. We observed the in Statistical Features we observed very crisis specific terms among the top ranked features. Also, we saw 7 different hashtags in top 50 features and that is indicative of how vocabulary specific important features are. As we analysed the semantic models, we saw more generic, yet crisis related concepts showing up in the top ranked features. Some of the semantics concepts existed across different crisis types.
• #33: Overall the Dbpedia semantics was the best performing model when the classifier was tested on an unseen type of the crisis. The avg. gain was 7.2% over the baseline. And it performed uniformly well across all the 3 tested types. The Dbpedia performed well, likely due to its better coverage and semantic depth. Something we need to explore more.
• #34: Few more take aways from the analysis were: generalizing the data semantically help in adapting new crisis events. Sometimes very broad and general concepts can result in underperformance of the classifier. For instance, virtue is the hypernym of diverse concepts which can often be used in different context. Also, sometimes semantic extraction can yield very unwanted concepts and in huge volumes. For e.g. here.
• #35: As a progression to this work, we need to perform a more in-depth error analysis. Currently we only take the type of event in account, which is broadly the nature of a given crisis. However, different events can have overlapping content based on similar situations. So it would make sense to take that also into account. The crisis data can also originate in different languages, so how the classifier can be tuned to handle multilingual aspect of the crisis data, that analysis should be expanded. We made an attempt on doing the same, and that can be referred to in the near future at the following research paper that we will present at upcoming semantic web conference.