[ASE2014] An Empirical Study on Reducing Omission Errors in Practice

Editor's Notes

• #2: Good afternoon, I’m Jihun Park, from KAIST, South Korea. This work is advised by Professor Miryung Kim at UCLA and Professor Doo-Hwan Bae at KAIST. Today, I will be presenting an Empirical Study on Reducing Omission Errors in Practice
• #3: Before I go into the main part of the talk, let me briefly summarize the motivation for our study. Over the past decade, many change recommendation systems have been proposed to identify additional change locations using change coupling, structural dependency, and cloning-based relationship. These approaches, however, only concentrated on predicting co-changed program entities. For example, given a program entity in a transaction, they predicted remaining entities of the transaction.
• #4: To study a real-world omission errors, we create a supplementary patch data set. We parse commit logs to get bug IDs corresponding to each revision, then we find that some bugs are fixed more than once. When a developer fixes the same bug multiple times, this means that the first commit is either incomplete or incorrect, requiring supplementary patches to fix the bug completely.  We define this type of error as omission errors. We also define the first commit as initial change, and the following commits as supplementary changes or supplementary patches. The goal of this paper is empirically studying the feasibility whether we can reduce these omission errors by predicting supplementary change locations given an initial change location.
• #5: Let me briefly describe our approach and key contributions, before I discuss in detail. To systematically investigate a real-world omission errors, based on the supplementary patch data set, we suggest a graph representation called Change Relationship Graph (CRG) based on program entities and the relationships between them. We found four observations. First, while a single prediction method is not enough, combining multiple prediction methods is inaccurate as well. Second, a machine learning technique based on past prediction accuracy information, boosting, does not significantly improve the prediction accuracy. Third, filtering prediction based on package or developer specific information do not improve the accuracy. Fourth, developers rarely make the same mistake on the same location. As a result, we would like to share our skepticism that reducing real-world omission errors is inherently challenging.
• #6: To represent the relationship between initial and supplementary change locations, we develop a graph representation called change relationship graph. Change relationship graph is defined by graph nodes and graph edges. We use program entities as graph nodes, such as classes and methods. Graph edges represents the relationship between the program entities. They include structural dependencies, such as extends, contains, and method invocations. Graph edges also include historical co-change, code clone, and name similarity relationships. We use Eclipse JDT core, Eclipse SWT, and Equinox p2 as our study subjects. Our goal is identifying the supplementary change location, give an initial change location.
• #7: As the first observation, We investigate the relationship between initial and supplementary change locations. We find that only 10 to 20% of supplementary change location can be connected with one edge from their initial change location. This result indicates that existing change recommendation systems can predict only a small portion of supplementary change locations. We use single or multiple edges as prediction rule. For example, a code clone edge and a calls edge can represent a prediction rule that suggesting candidate supplementary change locations that are connected by these two types of edges from the initial change location. Please note that there is an arbitrary node between the relationship edges, if it consists of multiple relationship edges. We identify every prediction rule connecting initial and corresponding supplementary change locations. We then investigate the accuracy of them when they are applied to the initial change locations to predict supplementary change locations. We found that the accuracies of the prediction rules are at most 10% in terms of f-score, which is very low. This result indicates that there is no repeated relationship between initial and supplementary change locations. A sub-conclusion here is that combining multiple traits does not predict supplementary change locations accurately.
• #8: For the second observation, We hypothesis that the past accuracy information may improve the accuracy of the future prediction. We divide our supplementary patch data set into a training set and an evaluation set. We use a boosting approach based on the accuracy information in the training set. Boosting is originally a machine learning technique, that combines weak learners to make a strong learner. It assumes that a method which showed high accuracy in the past, will also show high accuracy in the future. We calculate a prediction score by summing up trained accuracy of rules connecting initial and candidate supplementary change locations. For example, in this figure, we sums up trained accuracy of these two rules to get the prediction score for method 2. After that, we rank locations in the order of the prediction score, to suggest supplementary change locations. Our investigation result, however, showed that the precision of this approach is only 7% at most, even we suggest only three nodes as candidate supplementary change locations. We conclude here that boosting approach based on the past prediction accuracy also cannot accurately predict supplementary change locations.
• #9: As the third observation, We filter prediction rules based on package or developer specific information, because a similar pattern of relationship would be repeated in a specific package or by the same developer. For example, a specific relationship, code-clone relationship in this example, would be more common and more accurate in a specific package than the other relationships. We make boosting approaches, similar to the previous slide, based on package and developer specific prediction rules. We find that, however, the improvement are very negligible, or even the accuracy of the boosting approaches based on package or developer specific rules are even lower than the accuracy of a boosting approach based on regular rules in some cases. We find that no package or developer specific pattern between initial and supplementary change locations exists.
• #10: As the fourth observation, We investigate whether repeated patterns between initial and supplementary change locations exist. For example, if there is a pattern that initial change on file A implies a supplementary patch in file B, then we can suggest file B as the supplementary change location for the change on file A in the future. We investigate how many of the patterns are repeated in the version history. Our investigation result shows that the majority of patterns, 78% to 96% of them, appear only once in the version history. In addition, about 70 to 80% of initial change locations appear only once in the version history. These results indicate that developers rarely make the same mistake on the same location
• #11: In conclusion.. We systematically study omission errors using a real-world supplementary patch data set, and we also create a graph representation called change relationship graph based on the data set to represent the relationship between program entities. We found that version history based pattern mining cannot be accurate at finding supplementary change locations, because there is no repeated single relationship and combined relationships, and developers do not make the similar mistake again. In addition, past prediction accuracy information based machine learning technique, and filtering prediction based on package and developer specific information also do not improve the accuracy of the prediction. We would like to share our skepticism that reducing real-world omission errors is very difficult and inherently challenging.
• #12: Thank you for listening, I’m Jihun Park, from KAIST, South Korea. Thank you.