To Label or Not? Advances and Open Challenges in SE-specific Sentiment Analysis

Editor's Notes

• #3: My research focuses on human aspects in software, My contribution in the scope of the group focuses on human aspects in software engineering. In particular I leverage my background in affective computing, emotion recognition and natural language processing in the study of human aspects in software engineering both at an individual and at a team level.
• #5: In my reserachs I focus both on language analysis, using sentiment analysis on developers’ communication traces on several platforms, such as Github, Stack Overflow, Jira, etc. More recently, with our group we started investigating the role of emotion in software development by running user studies aimed at assessing the correlation between self-reported emotion and the perceived productivity during software development activities. Specifically, we are also building classifiers the detect positive and negative emotion as well as high or low emotional activation based on biometrics sensors that measure heart-related metrics, EEG that is brain activity, and galvanic skin response, that is the amount of electricity conveyed by the skin. 
• #10: This research fits in the vain of a research trend that emerged in recent years as researchers became interested in the use of sentiment analysis for software engineering applications. For example, correlation between emotions and productivity was investigated in issue tracking systems. Other studies investigate the possibility to implement early detection of anger and burnout to prevent undesired turnover. The role and impact of emotions in collaborative knowledge sharing was also studied on Stack Overflow. Also, mining of opinions about APIs from developer-generated content has been leveraged for building recommender systems. Similarly, in the field of requirements engineering, positive and negative sentiment in users’ feedback might be used to inform prioritization of development tasks by suggesting new features or enabling identification of bug based on the users’ complaints.
• #11: Early studies in this field, and when I say early I mean studies published about five years ago, use general-purpose sentiment analysis tools. Here are some example tools, for example NLTK has sentiment analysis libraries whose classification model were fine-tuned using movie reviews or tweets. Movie reviews were also used by the Stanford researcher in training their model based on deep-learning Other than supervised approaches, there are also tools relying on lexicon-based heuristics such as SentiStrength, wihch leverages a set of rule that compute the sentiment scores based on the presence of positive or negative sentiment words in the text. The performance of sentistrength were vaidated on six different datasets annotated manually and containing texts from social media, including Youtube comments, Twitter, my space texts, and so on.
• #12: Early studies in this field, and when I say early I mean studies published about five years ago, use general-purpose sentiment analysis tools. Here are some example tools, for example NLTK has sentiment analysis libraries whose classification model were fine-tuned using movie reviews or tweets. Movie reviews were also used by the Stanford researcher in training their model based on deep-learning Other than supervised approaches, there are also tools relying on lexicon-based heuristics such as SentiStrength, wihch leverages a set of rule that compute the sentiment scores based on the presence of positive or negative sentiment words in the text. The performance of sentistrength were vaidated on six different datasets annotated manually and containing texts from social media, including Youtube comments, Twitter, my space texts, and so on.
• #13: The reliability of general-purpose tools was questioned by benchmark studies showing how they perform poorly on technical texts and they disagree with each other, inducing a threat to conclusion validity for empirical software engineering studies. This is kind of expected as general-purpose tools are trained on corpora collected in different domains, such as movie reviews, newspaper, social media comments, and so on.
• #14: So why general-purpose tools fail? In search for an answers to this question, we manually investigated 800 questions, answers, and comments from Stack Overflow with specific focus on texts that were misclassified by SentiStrenght and found that the main cause of errors are the inability of tools to deal with contextual semantics, the fact that polarity of the lexicon might depend
• #19: Trying to overcome the limitations posed by general-purpose tools, researchers started to develop Software Engineering-specific tools for sentiment analysis. Two main approaches prevailed: on one hand, tools such as Senti4SD or SentiCR rely on supervised machine learning, combining different lexical features extracted from the raw text in a manually annotated gold standard. On the other hand, tools as SentiStrength-se and DEVA rely on lexicon-based approach, which means that they implement rules and heuristics based on the presence of positive and negative sentiment words as listed in sentiment vocabularies. So these tools cannot be retrained using a gold-standard.
• #20: We also developed our own tool for sentiment polarity detection, which you can download and re-train using your own custom gold standard dataset.
• #25: Misclassification of positive posts as negative occurs in 6.6% of the cases when the classification is performed with SentiStrength (see TABLE 9). This is what we consider a strong disagreement that should be avoided. Senti4SD reduces such misclassification to 2.4% of the cases. For example, sentences like ‘Is in so u need not worry! Internally the data is always stored as TEXT, so even if you create table with, SQLite is going to follow the rules of data type’ is erroneously classified by SentiStrength as negative due to the presence of negative lexicon (‘worry’) even if SentiStrength is supposed to correctly deal with negations (Thelwall et al. 2012) which should determine polarity inversion.
• #26: So, the question now is: given the lesson learned here, that platform-specific trained is required, the question is how many texts do we need to label to reliably retrain classifiers.
• #34: Better performance for supervised approaches
• #35: Better performance for supervised approaches
• #37: So, the question now is: given the lesson learned here, that platform-specific trained is required, the question is how many texts do we need to label to reliably retrain classifiers.
• #47: So, the question now is: given the lesson learned here, that platform-specific trained is required, the question is how many texts do we need to label to reliably retrain classifiers.
• #52: Manual inspection of texts misclassified by all tools reveal that polar facts are the most common cause of misclassification ad that they are mainly observed in ad hoc annotation datasets
• #53: So, the question now is: given the lesson learned here, that platform-specific trained is required, the question is how many texts do we need to label to reliably retrain classifiers.
• #54: And indeed, we found that tuning of tools for software engineering improves the classification accuracy. Specifically we observed that SE-specific tuning enhance the agreement of tools with manual annotation and with each-other
• #55: A question remains opens, which related to the off-the-shelf use of such tools. So the question we address in this study is whether we can use Software Engineering-specific tools for sentiment analysis off-the-shelf, without retraining them. Manual labeling is time consuming and gold standard datasets are not always available. So If I don’t have a manually annotated gold standard from the platform I want to study, can I safely reused these tools off-the-shelf, without retraining?
• #56: Specifically, we formulate three research question to check if tools agree with manual annotation and with each other in a cross-platform setting, that is in absence of a gold standard for retraining
• #57: We reused the Jira and Stack Overflow dataset we already leveraged in our previous benchmark and we specifically developed a third dataset with Github comments for the sake of this study. The Github dataset, of course, is available for reuse and can be downloaded from figshare. The three datasets are labeled based on the same emotion model, which is the Shaver model of emotions. In this model we map emotions as joy into positive polarity and emotions such as anger into negative polarity. Neutral polarity refers to the absence of emotions. The Stack Overflow and Github dataset are well-baleanced in terms of distribution of polarity lables while the Jira one include a prevalence of neutral texts. A difference in interrater agreement is also observed, with Github and Stack Overflow showing the highest Kappa values indicating substantial agreement among human raters
• #58: And of course we repeated this process for all couples of datasets, that is we trained three different models for each tools using the train set of each dataset and then we tested on the test set from each dataset, to simulate the off-the-shelf use of such tools. The setting on the diagonal correspond to the within-platform setting, that is when you have a gold standard from your own target platform and you can retrain.
• #59: The papers contains all detailed information regarding performance metrics. Here, for time constraint, I will only discuss f-measure. Let’s start from supervised tools and Github. Let’s assume your target platform in your study is Github and you have a gold standard for retraining. In a within-platform condition, we observe very satisfying performance with Senti4SD achieving .92 of f-measure and SentiCr .82
• #60: The situation changes when training is performed on a different platform: by training on Stack Overflow we observe a drop of .12 and .17 for Senti4SD and SentiCr, respectively. The situation is even worse when training on Jira.
• #61: Interestingly, we observe that in both cases the drop is reduced when Bag of Words are excluded from the feature sets. This is in line with the fact that the drop is higher for SentiCR that only relies on Bag of words, while Senti4SD also leverages semantic and lexicon-based features thus mitigating the drop and reducing overfitting to the training data. This provide evidence of the lower ability to generalize of such features in cross-dataset settings and suggest that positive and negative lexicon might be platform dependent. Furthermore, this indicates that shifts in lexical semantics might occur across platform due to platform-specific lingo or communication style.
• #62: This drop in performance is observed for all setting, thus reinforcing the intuition that the definition of domain, which is currently identified with the broad field of software engineering, should be probably narrowed down to the specific platform.
• #63: In particular we observe the highest drop when the Jira dataset is used for training. This indicate a higher ‘distance’ of the Jira gold standard from the other two, which might be potentially explained in several ways. First of all, the lexical distance we just discussed might be an explanation.
• #64: Also, this could be due to the difference in label distributions, which induce a bias towards the majority class for the Jira dataset, which is imbalanced in favor of the neutral class
• #65: Another explanation might be in the robustness of the gold standard in term of interrater agreement, which is substantial to almost perfect for Stack Overflow and Github. Furthermore, the labeling of these two datasets was coordinated by researchers at Collab research lab, at the University of Bari who hold a shared understanding of the conceptual model of emotions and who defined the guidelines for annotation according to their research goals.
• #66: As for the comparison between supervised and lexicon-based tools, we can see that in a within-platform condition supervised retraining enable achieving the best accuracy.
• #67: However, this is not necessarily true if the training is performed in a cross-platform setting, with lexicon-based tools outperforming supervised ones in some of the studied conditions, as in the case of Jira.
• #69: So, the question now is: given the lesson learned here, that platform-specific trained is required, the question is how many texts do we need to label to reliably retrain classifiers.
• #70: To answer this question, we retrained the supervised tools in a within-platfrom condition using incrementally bigger training sets and testing on the same 30% test partition used so far for the other experiments. Here are the learning curves for the GitHub and Stack Overflow dataset. We also report the performance of lexicon-based tools for reference, which of course do not change as we cannot retrain then. We observe that retraining is convenient compared to the performance of lexicon-based tools and a nearly optimal performance is obtained already with a minimal set of about 1,200 documents for Github and Stack Overflow
• #71: A different situation is observed for Jira, which is unbalanced. In this case retraining is beneficial only if a larger set of about 1600 document is available but the improvement over the lexicon based tool Sentistrength-SE is negligible. This might be due to the fact that SentiStrength-SE has been fine-tuned by the authors using subset of the Jira dataset we are using in this study. Another explanation might be the unbalanced polarity label distribution, which we didn’t manage to address successfully even when techniques as SMOTE are applied, as you can see from the bottom orange line in the chart.