CMPT470-usask-guest-lecture

Editor's Notes

• #2: Introduce yourself and the affiliation. Today I am going to talk about query suggestion for Concept location where we used Information Retrieval methods.
• #6: Introduce yourself and the affiliation. Today I am going to talk about query suggestion for Concept location where we used Information Retrieval methods.
• #7: This is a software change request. It has different sections like title, description and others. Now a developer’s task is to identify the most important terms and then use them for finding the source code to change.
• #8: To model the problem formally, this is a mapping problem. And the mapping is between concepts in the change request and the relevant source artifacts from the codebase. Our job is to identify the appropriate terms from the change request for the successful mapping.
• #10: There have been some studies on similar problem. However, most of these studies reformulate a given query. That means, the developer needs to provide an initial query first. But studies show that choosing that initial query itself is challenging. A study reported that only 12% of developers chosen search terms from the change request were useful. So, our focus is to choose the initial query from a change request rather than reformulation. The closely related work used a set of heuristics.
• #11: While the earlier work used heuristics for the same problem, we used Google’s PageRank algorithm for choosing the important terms from a body of texts. Here, the most important face in the crowd is the face everybody is looking at, right? This also goes true for world wide web. A page is reputed is it is referred by other reputed pages from the web. So, we adapt our search term identification after this model.
• #12: We identify search terms using two variants of PageRank--- They are called TextRank and POSRank in the information retrieval domain.
• #13: So, these are the pretty straight-forward steps of our approach. We take a change request, and perform standard NLP (stop word removal and splitting). We avoided stemming. Then from the pre-processed texts, we develop two types of graphs – text graph and POS graph. Then we derive importance score for each of the terms from those two graphs. Then we a do linear combination, perform ranking and choose the top words as the search terms based on their scores. Now, we will zoom in this sections more.
• #14: The idea behind this text graph is word co-occurrence. For example, these two terms—IResource and IJavaElement-- occur in the same context across multiple sentences. These are another two terms– element and reported—occur in the same context. Here we define context as a window size of two words within a sentence. We encode their co-occurrence into an edge in this text graph. This way, the whole change request can be converted into a text graph.
• #15: Similarly, we develop the second graph based on syntactic dependence among various parts of speech of sentence. We apply Jespersen’s Rank Theory of 3 ranks. More details on the paper. That is, some POS depends on others POS for their complete meaning. For example, verb modifies noun and adjectives from within the same sentence. We encode such dependencies into the connecting edge, and develop another text graph. Thus, some terms are more connected than others.
• #16: Now, we have two graphs developed from the change request based on two different dimensions --Word co-occurrence and syntactic dependence. Now, we apply the above algorithms adapted from PageRank for scoring. That is, a term’s importance will be determined by the importance of the surrounding terms, not just the connectivity. This is how Google beats the SPAM pages. We apply that in the case of concept location as well. This is the first time done in the concept location task, and this is our novelty.
• #17: So, this is how the score of a term is determined, based on the scores of the surrounding terms. That means, the score of Vi is determined based on the scores of Vj1 to Vj5. We collect scores for the terms from both graphs which we call TextRank and POSRank. We combine them, rank them and collect the top ones as the search terms.
• #18: For experiments, we select 8 subject systems from Apache and Eclipse. We collect 1939 change requests/bug reports from BugZilla and JIRA, and prepare the gold set by consulting the commit history of those projects from GitHub. For selecting bug fixing commits, we adopted the widely accepted approach. That is, we identify the Bug ID in the commit title, and then extract corresponding change set.
• #19: For experiments, We collect our queries and the baseline queries (e.g., title or description from the change request), and feed them to a code search engine. Then we collect their results/ranks and compare. For evaluation/validation, we used these four performance metrics.
• #20: Results show that our method can improve 52%--62% of the baseline queries, which is promising according to relevant literature. We consider various combinations as the baseline queries, and got similar performance. Our improvement and worsening ratios are significantly different according to statistical tests. The mean rank difference also shows that our mean ranks are closer to the top than the baseline.
• #21: In terms retrieval performance, precision and recall are not too high. Precision is close to 30% and the accuracy is close to 45% when Top-10 results are considered. But I guess, that has been the status quo for the last 15 years. So, nothing very dramatic. However, they are quite higher than the baseline performance actually.
• #22: When we extend the K-values, we found the accuracy is growing significantly. But, still, our performance remained higher than all the baselines. This shows the potential of our method.
• #23: We compared with two parallel methods– Kevic & Fritz used heuristics and the second is a classic query reformulation technique. While they were promising, but still our method beat them in all aspects, and the performance is significantly higher as you see.
• #24: If we see at box plots, we can see that our median metrics are significantly higher. While they relied to a set of heuristics and term weighting, our PageRank-based model seems to perform better.
• #25: When we consider various Top-K accuracy, we got similar findings. Our method located concepts correctly for 80% of the change requests whereas they did for 60% of them at best. This shows the potential of our technique.
• #26: You can simply read out the texts I guess.
• #27: Good morning, everyone. Introduce yourself. Today, I am going to talk about a query reformulation technique for concept location where we used an advanced term weighting method and performed machine learning.
• #28: Now, this is a real software change request. Here these two sections are important, and they contain information about the requested change.
• #29: Now when a request like this is submitted, a developer tries to find out important terms. Then they use those terms for finding the source code to change probably using a search engine like Lucene
• #30: That is, they try to map the concepts discussed in the change request to appropriate source code sections like this. This is how, the term comes– “concept location” if you want me to define it.
• #31: But this concept location is NOT an easy task. For example, these two very reasonable queries from the change request do not perform well. This second one returns correct results at 77th positions, which is not acceptable of course. So, what is needed here is– the reformulation of the query for better. Now, there are traditional tool supports for doing that. What most of them do is, they throw in the initial query to the search engine, collect the results, and then collect most important terms from those results for The reformulation of the initial poor query.
• #32: Now, these are the reformulated queries from three existing such methods. Now, they did some improvements in the ranking, and return results a bit closer to the top. But as you can see, they are not clearly enough. Developers want the results at the top positions, so they are still costly for practical use.
• #33: Now, we investigate this part of the reformulation process, and found that Most of the existing techniques are using this equation for determining importance of a term. That is, they are selecting TF-IDF to find the words for query reformulation. In other words, they are relying on the frequency of a term as a proxy to its importance.
• #34: Now, this is a metric which has been on the play from last the century. It was proposed in the 70s. It is a good metric, but it was actually proposed for regular texts such as news articles. On the other hand, we are dealing with source code here. Now, regular texts and source code have different semantics and different structures. They are not the same So, metrics for regular texts are not appropriate for the source code– this is our hypothesis.
• #35: So, we made two contributions here. We propose CodeRank– a novel and appropriate term weighting method for source code. We propose ACER -- a novel query reformulation technique that uses this term weight.
• #36: First comes CodeRank.
• #37: Now, what we did? We extract important artifacts from source code such as method signature, formal parameters and field signatures from the code. We mostly used AST parsing and regular expressions for this. The idea is – signatures capture more rich intent than other texts. For example, method signatures provide the intent whereas the method body implements the intent with lots of noise.
• #38: Now once such items are extracted, we split them. Now as we see, these single terms share some kind of semantics to convey a broader semantic. That is, they complement each other in this context. Now, we capture such semantic dependencies in the source code, and develop a term graph like this.
• #39: Now, once the graph is developed, we use a popular graph-based algorithm called PageRank for determining the node importance. OK Lets go visual. In a crowd, the most important person is the one whom everybody is looking at. It can be also seen as votes. The person who is voted the most is the leader. We also follow that concept in the context of our term graph. That is, the term which is connected the most with other terms is an important term. Now, this scoring is a recursive process, we finally get a ranked list of important terms which can used as reformulation terms.
• #40: Now comes the ACER, the second contribution.
• #41: This is the schematic diagram of our approach. So far we talked about these parts of our approach. Now we will zoom in this part.
• #43: Once the CodeRank is calculated, we collect multiple reformulation candidates for a given initial query. As we discussed, a source document has various contexts– method signature, field signature and so on. We make use of such contexts, and develop multiple reformulation candidates. Now, since we have multiple options, we have to choose the best reformulation. In order to do that, we apply machine learning. In particular, we determine the quality of each candidate using 20 quality metrics that mostly came from IR domain. Then we use a regression-tree based classifier and suggest the best reformulated query.
• #44: Now lets see what is the outcome. Here, we have created three reformulation candidates using CodeRank and source document contexts. Then our ML classifier returns the best option, and it returns the result at the 2nd position. Now, if we look closely, our technique identifies two unique terms which made the real difference in performance.
• #45: For experiments, we select 8 subject systems from Apache and Eclipse. We collect 1.5 thousand change requests/bug reports from BugZilla and JIRA, We use the report title as our query and prepare the gold set by consulting the commit history of those projects from GitHub. These are the widely accepted approach to do experiments in this area.
• #46: For experiments, We collect our queries and the baseline queries , and feed them to a code search engine. Then we collect their results/ranks and compare. For evaluation/validation, we used these four performance metrics.
• #47: Now, in our experiment, we answer these five research questions.
• #48: In the first research question, we compare our queries with the baseline queries. As we see, method signature based reformulation performs the best than the other two options. However, the Machine Learning selects the best among the three, and provides the best performance. For example, our reformulation improves 71% of the queries, preserves 26% and degrades only 03% of the queries. So, obviously, we are improving more queries than degrading.
• #50: In the second research question, we compare CodeRank with the traditional term weights – Term Frequency and TF-IDF. We see that TF performs better than TF-IDF, which is interesting. Anyway, when compared with our CodeRank, we see that TF performs better initially but then CodeRank outperforms it later., especially for 10-15 reformulation terms. That is, few highly frequent terms are really important, but yes, CodeRank is more reliable than Term Frequency for term importance.
• #51: In the third research question, we show how document structures/contexts make a difference. These are the number of improved queries by various reformulation candidates. Now we see 19% of the total improvement are unique to each single contexts. That is if, we consider only method signatures for query reformulation, we miss the improvements made by field signature based reformulations. Again, if we consider the whole texts rather than signatures, we also miss some query improvements. This is not only for CodeRank, this is also true if we employ term frequency in those contexts. Thus, document contexts matter for query reformulation.
• #52: Now, when we consider query improvements by ACER and Term frequency in terms of Vern diagram, We also found that 66% overlaps, but ACER provides a unique set of improvements which is three times that of TF. Now ACER does document structures and TF does not. And we see the difference here.
• #53: In the fourth research question, we do the calibration for reformulation length. We found the best performance is achieved when the reformulation length is between 10 to 15. This where CodeRank saturates.
• #54: In the fifth research question, we compare our query improvement and worsening ratios with the existing methods. We see our median improvement is much higher than others. More importantly, we degrade a very low amount of queries compared to the others. Obviously, these measures are significantly higher. Thus, according to our investigation, ACER is the winner. But we must also admit, the ML-based approach is less scalable, and now we are working on the tool.
• #55: Thus, these are take-home messages. Query reformulation is a challenging task for the developers. Google does not work local source code repository. Traditional term weights are not clearly sufficient or appropriate for source code. We provide CodeRank, a novel term weight for source code. We provide ACER, an improved reformulation technique. Our technique improves about 71% of the queries and degrades only a handful queries. Comparison with the state-of-the-art shows the promising aspect of our method.
• #56: Thanks for your time and attention. I am ready to have a few questions.
• #57: When we consider various Top-K accuracy, we got similar findings. Our method located concepts correctly for 80% of the change requests whereas they did for 60% of them at best. This shows the potential of our technique.
• #58: We tried with source code and Stack Overflow to look for semantically similar words. What’s next?