On the Unreliability of Bug Severity Data

Editor's Notes

• #2: Thanks prof. z for the introduction, thanks all for taking time to attend this talk, I am honored to be invited here. As you can see from the title, I’d like to share some of my experience in dealing with inconsistent labels, which is important for making your later analysis on the data reliable and effective. Please feel free to interrupt if you find any difficulties regarding the slide.
• #3: To begin with, I would like to introduce supervised machine learning models, which are probability the most common machine learning models used nowadays. And they are also the ones that are affected the most by the quality of labels. In the slide, we see a general flow of supervised machine learning, it takes labelled training data as input, gleaning information from it, and eventually learn a model that can label new data. Let’s me give you a simple example here, at the top of the slide, you can see 7 images, each of them is associated with a label 1/0 indicating whether there is a cat in the image. Supervised machine learning takes these data-label pairs as training input, and then extracts features from the images. In traditional machine learning flow, features are defined manually, while in the latest deep learning techniques, features are learned from the data. After the feature extraction process, a model is learned for a mapping between feature values and labels, so that given the new image shown at the right bottom of the slide, we hope that the learned model is able to identify that there is a cat in the image.
• #4: Since supervised machine learning models are popular, and rely heavily on labels, intuitively, we machine learning practitioners want clean labels in our training data. However, things don’t always go as we wish. Have you noticed that just among the 7 images we saw in the previous slide, the image cycled in red contains a dog, instead of a cat, in this case, we say we encounter an incorrect label. In fact, real-world data often contain noisy labels due to various reasons. For example, we human make mistakes, including experts. secondly, as collecting reliable labels is a expensive and time costly task, many studies leverage crowdsourcing platforms to collect non-expert labels in a cheap and fast way, and some regard machine generated labels as the ground truth. Last, noisy labels might simply due to communication or encoding problems.
• #5: In fact, people have studied the impact of noisy labels for a long time in the machine learning areas, theoretically or empirically demonstrate that noisy labels can bring negative consequences for learning. The most important two includes descried performance and unreliable performance measures. The image on the right side shows an study regarding performance of traditional classifiers on a particular task, we could see that the performance of the classifiers all drop dramatically after noise level is greater than 3%. Noisy labels often lead to more complex model, and incorrect influential features. http://www.stat.purdue.edu/~jianzhan/papers/sigir03zhang.pdf
• #6: In fact, people have studied the impact of noisy labels for a long time in the machine learning areas, theoretically or empirically demonstrate that noisy labels can bring negative consequences for learning. The most important two includes descried performance and unreliable performance measures. The image on the right side shows an study regarding performance of traditional classifiers on a particular task, we could see that the performance of the classifiers all drop dramatically after noise level is greater than 3%. Noisy labels often lead to more complex model, and incorrect influential features. So the message I want to deliver here is, we should take care of noisy label when we design machine learning models.
• #7: In most of the research on analysing noisy labels, we have an important assumption that absolute ground truth for label exist, like we could easily tell that the cycled image is wrongly labelled. However, classification can be subjective, where the ground truth is hard to tell.
• #8: For instance, two doctors may give different diagnosis regarding the same patient based on their experiences, especially when the information are not fully collected. In the field of computer security, different companies have their own standard in determining whether a software is malware or not. Thus when people combine different malware benchmarks together, there might be conflict labels on the same software. In the image tagging process, users are allowed to create tags by themselves, thus when we may find different tags regarding the same object. Other data such as movie ratings and application ratings also encounter inconsistent caused by the subjective classification process.
• #9: To summarize the inconsistent labels noise we have seen so far, we can divide them into two groups depend on whether the label can be represented using a numeric variable or categorical variable. For questions regarding opinions of people,like interestingness of a book, a score between 0-10, or 1-5, is usually assigned to measure levels of agreement on a statement. So we can use a numeric value to represent each category. For scenario like image tagging, each tag is a categorical label. Similarity, job titles created by different companies may be different for the same person, or the same job. There are no standard terminology regarding the same data. Since subjective classification tasks often happens, especially when we want to model user behaviours and preference, and if we just ignore the inconsistent label noise introduced by the process, the learning will suffer from performance drop, and many other negative consequences as we have talked in the beginning of this talk.
• #10: So here comes the question, how people cope with inconsistent label noise? Well, to answer this question, we need to first figure out when the inconsistent labels are encountered. Sometimes, we are the ones who can control the label collection process, sometime, we start with labelled data. In the label collection process, some people do not aware of potential inconsistency introduced by the subjective nature of the classification task. While some people do care about the labeling process, especially when they are creating benchmark dataset. Several strategies are adopted, which I believe all researchers should consider before collecting human annotated labels. For example, involving multiple labelers and making sure that one instance has been labelled more than one labelers. If disagreement appear regarding one instance, we should consider either throw the data, or resolve the disagreement among labelers. Recently, there is also a trend of adopting pairwise comparison rather than assigning an absolute score, but this method would require many pairs of comparisons. If we do not have control on the process, but we have labels provided by multiple labelers, many studies go for different voting methods to merge multiple labelers’ into one label, but this method … Other studies will study the reliability of each labeler and filter outliers, or treat label as a distribution over all possible labels, which is called uncertain label. But how about we only have one label per instance, especially when we know that nothing has been controlled in the label collection process. In the literature, rare work consider this case, but we keep seeing the danger of ignoring such inconsistent noise, which motivate my work on this area. The key challenges for single labeler case, are: How to measure inconsistency of labelers? How to cope with such inconsistency?
• #11: And the key point of our solution is to leveraging collectively provided labels, which I call community intelligence on other tasks to transfer single labeler setting into multi labeler settings.
• #12: This work is part of my long-term research program which focuses on coping with data quality issues in big data settings.
• #13: bug severity level reflects the impact of bug on the system, it is assigned during bug reporting process, which is shown in the slide.