Extracting Code Segments and Their Descriptions from Research Articles
1 like119 views
This paper presents a technique for automatically extracting code segments from research articles and associating them with descriptions in the surrounding natural language text. The technique uses heuristics to identify "seed" sentences that likely contain or reference code, and then examines neighboring text to find descriptions. An evaluation on 100 code segments from papers showed a precision of 68% and recall of 21% for identifying code descriptions. The authors aim to improve accuracy and scale their approach to benefit tasks like code recommendation and documentation generation.
Extracting Code Segments and Their Descriptions from Research Articles
- 3. Bug Reports Emails Blog Posts Q & A forums Code Reviews Documentation E-books Research Papers Course Materials Presentations Public Chats Benchmarks Research Papers DL Domain # of articles ACM DL Computer Science > 300,000 IEEE Xplore Computer Science > 3,500,000 DBLP Mostly Computer Science > 3,729,582 https://en.wikipedia.org/wiki/IEEE_Xplore https://cacm.acm.org/magazines/2011/7/109905-acm-aggregates-publication- statistics-in-the-acm-digital-library/fulltext http://dblp.uni-trier.de/ 3
- 7. Excerpt from a research article To understand the difficulty of fixing a memory leak, let us take a look at an example program in Fig. 1. This is a contrived example mimicking recurring leak patterns we found in real C programs. Procedure check_records checks whether there is any bad records in a large file, and the caller could either check all records, or specify a search condition to check only part of records. In this example, both get_next and search_for_next will allocate and return a heap structure, which is expected to be freed at line 12. However, the execution may break out the loop at line 10, causing a memory leak. 7 Code segments [Bacchelli et al. (ICPC’10) Tang et al. (KDD’05) Bettenburg et al. (MSR’08) Subramanian et al. (MSR’13) Rigby et al. (ICSE’13) Natural language text describing each code segment
- 13. Identify Seeds: References Code By Position 13 Fig.5 shows a typical test method of this pattern. The method tests a set of basic functionality of API class BasicAuthCache, including the method put, get, remove and clear. There are three test scenarios in the method: line 4-5, line 6-7, line 8-10. They share two data objects, cache and authScheme. Their method invocation sequences are not same and there is no unified test target method. But there is a common subsequence among three method invocation sequences, i.e., the invocations of get and HttpHost. This code snippet obtains a user name (user- Name) by invoking request.getParameter(“name”)and uses it to construct a query to be passed to a database for execution (con.execute (query)). This seemingly innocent piece of code may allow an attacker to gain access to unauthorized information: if an attacker has full control of string userName obtained from an HTTP request, he can for example set it to 'OR 1 = 1;- -. Two dashes are used to indicate comments in the Oracle dialect of SQL, so the WHERE clause of the query effectively becomes the tautologyname = ' ' OR 1 = 1. This allows the attackerto circumvent the name check and get access to all user records in the database.
- 14. Fig.5 shows a typical test method of this pattern. The method tests a set of basic functionality of API class BasicAuthCache, including the method put, get, remove and clear. There are three test scenarios in the method: line 4-5, line 6-7, line 8- 10. They share two data objects, cache and authScheme. Their method invocation sequences are not same and there is no unified test target method. But there is a common subsequence among three method invocation sequences, i.e., the invocations of get and HttpHost. ReferencesCodeFigure Score=3 ContainsCodeIdentifiers Score=2 ContainsCodeIdentifiers Score=2 ReferencesCodeByPosition Score=2 ContainsCodeIdentifiers, Score=2 ReferencesCodeByPosition Score=2 TextBefore Score=1 14 Identify Seeds: Putting It All Together *****Code Segment appears here***** • Scoring sentences • Equal • Accuracy-based • Threshold Analysis Heuristic References CodeFigure Contains Code Identifiers References Code ByPosition Text Before, Text After Score 3 2 2 1
- 15. Identifying Neighboring Code-related Text • Heuristic 1: At least 1 sentence is a seed • Heuristic 2: At least (25%, 50%, or 75%, respectively) sentences in the paragraph are seeds Fig.5 shows a typical test method of this pattern. The method tests a set of basic functionality of API class BasicAuthCache, including the method put, get, remove and clear. There are three test scenarios in the method: line 4-5, line 6-7, line 8-10. They share two data objects, cache and authScheme. Their method invocation sequences are not same and there is no unified test target method. But there is a common subsequence among three method invocation sequences, i.e., the invocations of get and HttpHost. 5 out of 6 (75%) sentences are seeds 15 Heuristic 1 Heuristic 2 whole paragraph is a description
- 16. Evaluation Methodology • Research Question: How effective is our approach to automatically identify code descriptions in natural language text of research articles? • Subjects: 100 code segments from ACM DL and IEEE Xplore journal and conference software engineering papers • Gold Set: • 10 Human annotators (non-authors) • Measures: • Overall code description identification: Precision and recall • Seed identification: Precision 16
- 17. Evaluation Results Minimum # of Seeds Precision Recall 1-24% 39.05 70.20 >= 25% 53.41 50.33 >= 50% 66.04 28.45 >= 75% 68.30 20.53 Overall system effectiveness 17
- 20. Main Threats to Validity • Unable to distinguish between pseudocode and code fragments • Papers with no pseudocode, plan to extend the approach to identify both. • Evaluation relies on human judges • Human judges with experiences in programming and research paper reading. • Each code segment judged by at least two judges. • Scaling to extensive evaluation set might lead to different results • Plan to expand the evaluation with more participants, and research papers containing more code segments. 20
- 21. Related Work • Analyzing Collections of Research Articles: Cruzes et al. (ESEM’07), Siegmund et al. (ICSE’15) • Code Segment Extraction: Bacchelli et al. (ICPC’10), Tang et al. (KDD’05), Bettenburg et al. (MSR’08), Subramanian et al. (MSR’13), Rigby et al. (ICSE’13) • Code Description Identification: Bug Reports: Panichella et al. (ICPC’12), Q&A: Vassallo et al. (ICPC’14), Wong et al. (ASE’13), Rahman et al. (SCAM’15) 21
- 22. Summary • Automatically identify & map text describing code segments in research articles • CoDesNPub Miner outputting XML-based representation associating code segments with their descriptions • Evaluation of the effectiveness of code description identification techniques • Precision = 68% • Recall = 21% 22 Future Work Improve recall and precision Fully automate preprocessing Expand the experiments