RACK: Code Search in the IDE using Crowdsourced Knowledge

Editor's Notes

• #2: Introduce yourself + introductory statements. Today, I am going to talk/demonstrate --- how code search can be performed using natural language queries where we used crowd sourced knowledge from Stack Overflow.
• #3: Currently, we have several code search engines such as Black Duck, Krugle, GitHub code search and so on. But traditional code search engines generally do not perform well with natural language queries. For example, Black Duck does not return any results for this simple query—”how to send an email in Java” Its because these engines do simple keyword matching.
• #4: Here, we see Krugle and GitHub return some results for the same query. But they also use simple keyword matching. These keywords can match from code tokens or code comments, as we can see. Besides, there is a larger problem with keyword matching would be--vocabulary mismatch problem. That is, if query and source code have different vocabularies, keyword matching will not work.
• #5: Now, with the web search engines, the same query returns thousands of results. But the bigger problem is, you cannot guarantee that they will contain relevant code examples. Developers generally like to play with working code examples rather than dealing with lots of texts. So, there is information overload for the developers to deal with in this case.
• #6: So, we are trying to solve this research problem: Code example search using natural language queries. One obvious solution is to translate the NL query into relevant API names, and then use them for the code search. Now, the challenge is how do you get the relevant API names for a particular code search?
• #7: Now lets see how can we do that. In Stack Overflow, users often ask questions like this—”How to generate MD5 hash?” Then someone expert answers the question that often contains the relevant API names. For example, here we see that MessageDigest is a relevant API for performing the task. More importantly, such information/answer is validated by a large developer crowd who agree with the answer. We mine thousands of questions and answers like this, and then translate a natural language query into relevant API names.
• #8: Now this idea comes with several challenges, and we need to solve them first before using information from Stack Overflow. For example, we have to answer these questions: First, do the accepted answers in Stack Overflow use API names frequently, as a part of the solutions? Or this is just occasional? Second, what percentage of the APIs are mentioned in those accepted answers from Stack Overflow? Third, do the titles from Stack Overflow questions overlap with code search keywords?
• #9: These are the findings for our RQ1. We determine API class frequency per accepted answer, and analyze their distribution. The frequencies have a heavy-tailed distribution such as Poisson distribution. And the mean frequency per answer is 2.37 with a 95% confidence interval between 2.33 and 2.41. That means, at least 2 API classes are used in each of the accepted answers of Stack Overflow.
• #10: These figures show how core Java API packages are covered by SO answers. We consider 11 core Java packages from standard Java libraries. We see that on avearge, more than 60% of the API classes are covered, i.e., used by questions and answers of Stack Overflow. This is a significant amount. More granular details are in the main paper. However, these findings suggest that SO is a good source for collecting API related information.
• #11: RQ3 focuses on the overlap between SO question titles and the real life search queries collected for our study. Fig. 6 shows the percentage of title tokens that matched with real life query tokens for the last 8 years. Queries are taken from the Google search history of the first author. This is above 70% on average which is really interesting.
• #12: So, these are summary of our exploratory findings. You can read them out, then say Stack Overflow is a good source for relevant API names for our query reformulation.
• #13: Now, we have these three steps in our methodology. (1) We first mine 344K questions and answers from Stack Overflow, and construct a Token-API mapping database. (2) Then we reformulate the NL query into relevant API names using two novel metrics. The details are in the paper. (3) Then we search for code examples in GitHub using those relevant API names.
• #14: The first step is Token-API mapping database development. This database provides a mapping between natural language tokens and relevant API names with the help of crowdsourced knowledge. We first collect the accepted answers of Stack Overflow questions, and extract their code segments or code like elements. Then we parse them for API class names using regular expressions, and collect them. Then we also collect the titles from SO questions, and perform standard natural language processing on them to generate unique tokens. So, now we have a list of tokens and a list of API classes from each Q & A thread, and they co-occurred due to semantic relevance. This co-occurrence is derived from the knowledge/experience of the crowd. So, we link those tokens and APIs, and store them in a database, and we call it Token-API mapping database.
• #15: We have the database. Now, the next step is pretty much straight forward. We collect the tokens from the query, and collect candidate APIs from Token-API mapping database using two co-occurrence based heuristics. Please consult the paper for metric details. This returns a ranked list of relevant API names for the NL query.
• #16: Now, we apply that relevant API names to GitHub search API which returns a list of relevant source code files. Now, we do AST parsing on them, extract the method implementations, and apply textual matching between query and code segments. Please note that now the query is relevant API names. Then based on relevance, we return a list of code snippets.
• #17: Experiment. We evaluate and validate the query reformulation part of our technique, and the code search part is basically leveraging the an existing search API. We conducted experiments using 150 code search queries, and they are collected from three programming tutorial websites– KodeJava, Java2s and Javadb.com Relevant past studies also used the same sites. In particular, we select the questions related to programming, and consider the question title as the query. They are quite similar to real life code search queries. Then we analyze the code example as well as the API classes from corresponding answers, and consider the APIs as the gold set for the query. In average, we select 3-5 API classes as the gold set API for each query
• #18: This is the performance of our technique. For Top-5 recommendation, it provides about 63% Top-K accuracy, and for Top-10, it provides about 79% accuracy which are quite promising. The mean average precision is 35%, and the recall is 45%. Although, the precision is a bit low, the technique’s main strength is its capability to return APIs for a wide number of queries. Because, the question titles provide a very wide vocabulary from thousands of people, and about 70% of the standard Java APIs are discussed in Stack Overflow. So, it would possibly always provide a better accuracy than other competitive techniques.
• #19: We also compare with two baseline techniques adapted from Thung et al., the state-of-the-art from ASE 2013. We see that the our technique is performing relatively better. It has a bit larger variance, but still median performance is higher than the other 2 baseline techniques. The detailed evaluation can be found in the main paper.
• #20: Now, this is the UI of our tool.
• #21: Lets see the demo of our code search tool, RACK. Note: Adding the offline video misses the annotation feature of youtube videos. That annotation can be only seen from youtube domain. So, please show the demo directly from the youtube.
• #22: So, these are take home messages. You can read them out.
• #23: Thanks for your attention! Questions?
• #24: We use two heuristics to select the candidate APIs for a search query. The first one is– Keyword—API Co-occurrence (KAC). That means we consider how frequently a search keyword and an API co-occurred across various questions and answers from Stack Overflow. This co-occurrence can be either due to semantic relevance or by chance. We discard the random association using a threshold frequency, but mostly such co-occurrence is due to some kind of relevance. So, for each keyword, we would collect a candidate list. Then they would be sorted and ranked. From that rank, we extract a heuristic score for each candidate API. We call this score--KAC score.
• #25: The second heuristic considers another aspect– Keyword—Keyword Coherence That is, the two candidate APIs can be frequent with two different keywords. But these two APIs should be coherent with each other as well. If they are not coherent/compatible, then they cannot solve a technical problem. So, we extract a coherent API list using the contextual word list for each pair of query tokens. We then determine cosine similarity between context of the two tokens which can be considered as the semantic relevance between these tokens. Then that score is considered as the coherence score for the selected API list.