Fighting Software Inefficiency Through Automated Bug Detection
0 likes258 views
The document discusses automated detection of software bugs and inefficiencies. It begins with background on the importance and costs of software bugs. The speaker then discusses their research into detecting performance bugs through pattern matching and developing generic detectors like Toddler and Caramel. Toddler uses dynamic analysis to detect inefficient nested loops based on repetitive memory access patterns. Caramel is a static detector that finds inefficient loops amenable to simple fixes involving condition breaks. Evaluation on various applications found over 150 new bugs, most of which were fixed by developers. The work represents different aspects of automated bug detection and recovery.
![Figh%ng so*ware bugs is crucial
• So3ware is everywhere
– h5p://en.wikipedia.org/wiki/List_of_so3ware_bugs
• So3ware bugs are widespread and costly
– Lead to 40% system down Eme [Blueprints 2000]
– Cost 312 Billion lost per year [Cambridge 2013]
6](https://image.slidesharecdn.com/shan2016mentor-160629165052/85/Fighting-Software-Inefficiency-Through-Automated-Bug-Detection-6-320.jpg)
![Bug detec%on examples
– Memory bug detecEon
• Pa5ern: over-bound writes, …
• DetecEon: check memory accesses & …
– Concurrency bug detecEon
• Pa5ern: data races, atomicity violaEons, …
• DetecEon: check memory accesses & …
9
P = malloc (10);
P[100] = ‘a’;
if (P)
*P=‘a’;
P=NULL;](https://image.slidesharecdn.com/shan2016mentor-160629165052/85/Fighting-Software-Inefficiency-Through-Automated-Bug-Detection-9-320.jpg)
![An Empirical Study of
Real-World Performance Bugs
Are there performance bugs?
Are they important?
What types of performance bugs are there?
12
Understanding and detecting real-world performance bugs [PLDI '12]](https://image.slidesharecdn.com/shan2016mentor-160629165052/85/Fighting-Software-Inefficiency-Through-Automated-Bug-Detection-12-320.jpg)
![Bug Examples
15
+ int i = -k.length();
- while (s.indexOf(k) == -1) {
+ while (i++<0 ||
+ s.substring(i).indexOf(k)==-1)
{s.append (nextchar());}
Patch for Apache-Ant Bug 34464
for (i = 0; i < tabs.length; i++) {
…
tabs[i].doTransact();
}
+ doAggregateTransact(tabs);
Mozilla Bug 490742 & Patch](https://image.slidesharecdn.com/shan2016mentor-160629165052/85/Fighting-Software-Inefficiency-Through-Automated-Bug-Detection-15-320.jpg)
![Sta%c inefficiency pa=erns exist
• StaEcally checkable inefficiency pa5erns exist
+ int i = -k.length();
- while (s.indexOf(k) == -1) {
+ while (i++<0 ||
+ s.substring(i).indexOf(k)==-1)
{s.append (nextchar());}
Patch for Apache-Ant Bug 34464
for (i = 0; i < tabs.length; i++) {
…
tabs[i].doTransact();
}
+ doAggregateTransact(tabs);
Mozilla Bug 490742 & Patch](https://image.slidesharecdn.com/shan2016mentor-160629165052/85/Fighting-Software-Inefficiency-Through-Automated-Bug-Detection-18-320.jpg)
![Toddler
A dynamic and generic detector
targeEng inefficient nested loops
22
Toddler: Detecting Performance Problems via Similar
Memory-Access Patterns [ICSE '13]](https://image.slidesharecdn.com/shan2016mentor-160629165052/85/Fighting-Software-Inefficiency-Through-Automated-Bug-Detection-22-320.jpg)
![Caramel
A staEc and generic detector
targeEng inefficient loops with
simple patches
30
CARAMEL: Detecting and Fixing Performance Problems That
Have Non-Intrusive Fixes [ICSE'15]
Won SIGSOFT Distinguished Paper Award](https://image.slidesharecdn.com/shan2016mentor-160629165052/85/Fighting-Software-Inefficiency-Through-Automated-Bug-Detection-30-320.jpg)
![Work from my group
40
In-house
bug detecEon
[ASPLOS06];
[SOSP07];
[ASPLOS09];
[ASPLOS10];
[ASPLOS11];
[OOPSLA13]
[PLDI12];
[ICSE13];
[ICSE15]
In-field
failure recovery
[ASPLOS13.A]
[FSE14]
Not yet
In-field
failure diagnosis
[OOPSLA10];
[ASPLOS13.B];
[ASPLOS14];
[OOPSLA16*]
[OOPSLA14]
In-house
bug fixing
[PLDI11];
[OSDI12];
[FSE16]
[CAV13]
concurrency
bugs
performance
bugs](https://image.slidesharecdn.com/shan2016mentor-160629165052/85/Fighting-Software-Inefficiency-Through-Automated-Bug-Detection-40-320.jpg)
Fighting Software Inefficiency Through Automated Bug Detection
- 3. My life J 3
- 8. How to detect bugs? • Study & understand real-world bugs • Discover pa5erns of common bugs – Source code level – Binary code level – … • Design pa5ern-matching program analysis – StaEc analysis – Dynamic analysis – … 8
- 9. Bug detec%on examples – Memory bug detecEon • Pa5ern: over-bound writes, … • DetecEon: check memory accesses & … – Concurrency bug detecEon • Pa5ern: data races, atomicity violaEons, … • DetecEon: check memory accesses & … 9 P = malloc (10); P[100] = ‘a’; if (P) *P=‘a’; P=NULL;
- 11. How did this start? • One of our bug detectors is strangely slow – Why not profiling? • Lots of noises in profiling • Measuring cost not inefficiency • Am I able to do this? • Where should I start? 11
- 12. An Empirical Study of Real-World Performance Bugs Are there performance bugs? Are they important? What types of performance bugs are there? 12 Understanding and detecting real-world performance bugs [PLDI '12]
- 14. Findings • Are there performance bugs? – Yes • Are they important? – Some are • What types of performance bugs are there? – What are their root causes? – Where are they typically located? – How are they usually fixed? 14
- 22. Toddler A dynamic and generic detector targeEng inefficient nested loops 22 Toddler: Detecting Performance Problems via Similar Memory-Access Patterns [ICSE '13]
- 26. How to detect? • Input: Test code + system under test • Steps: 1. Instrument the system under test Monitor loop starts/ends & memory reads inside loops 2. Analyze trace produced by instrumentaEon IdenEfy repeEEve memory-read sequences • Output: Loops that are likely performance bugs 26
- 27. Evalua%on Subjects and New Bugs 27 Applica%on Descrip%on LOC Known Bugs New Bugs Fixed Confirmed Ant Build tool 109,765 1 8 1 0 Apache CollecEons CollecEons library 51,516 1 20 10 4 Groovy Dynamic language 136,994 1 2 2 0 Google Core Libraries CollecEons library 156,004 2 10 1 2 JFreeChart Chart framework 64,184 1 1 0 0 Jmeter Load tesEng tool 86,549 1 0 0 0 Lucene Text search engine 320,899 2 0 0 0 PDFBox PDF framework 78,578 1 0 0 0 Solr Search server 373,138 1 0 0 0 JDK standard library 2 0 0 JUnit tesEng framework 1 1 0 9 Apps + 2 Libs 50,000 – 320,000 11 44 15 6
- 28. Toddler vs. HProf 28 Known Bug Bug Detected? False P. Rank Slowdown TODD. PROF TODD. PROF TODD. PROF Ant ü û 0 19.3 13.7 4.2 Apache CollecEons ü ü 0 1.0 10.0 2.1 Groovy ü ü 0 3.7 15.5 3.7 Google Core Libraries #1 ü ü 0 1.8 9.0 3.8 Google Core Libraries #2 ü û 0 5.3 7.5 3.2 JFreeChart ü û 0 53.7 13.4 8.8 JMeter ü û 0 10.3 8.5 1.9 Lucene #1 ü û 0 7.7 6.8 2.5 Lucene #2 ü ü 0 3.1 25.4 3.1 PDFBox ü û 1 18.8 51.8 12.1 Solr ü û 0 178.3 114.2 7.1 11 11 4 1 n/a 15.9X 4.0X
- 30. Caramel A staEc and generic detector targeEng inefficient loops with simple patches 30 CARAMEL: Detecting and Fixing Performance Problems That Have Non-Intrusive Fixes [ICSE'15] Won SIGSOFT Distinguished Paper Award
- 31. What are perf. bugs not fixed? 31 Correctness Maintainability Manual effort Potential speedup under certain workload Can we detect bugs with simple fixes?
- 32. What is the pa=ern? • What is a typical simple fix for an inefficient loop? • What types of bugs have the above type of fix? – We thought for a long Eme … 32 for(…) + if (cond) break;
- 34. What Bugs Have CondBreak Fixes? 34 Every Itera%on Late Itera%ons Early Itera%ons No-Result Type 1 Type 2 Type Y Useless-Result Type X Type 3 Type 4 Where Is Computation Wasted?How Is Computation Wasted?
- 38. Evalua%on Subjects and New Bugs • 15 applica%ons – 11 Java, 4 C/C++ • 150 new bugs • 116 bugs fixed – 51 in Java – 65 in C/C++ • Only 4 rejected • 22 bugs in GCC fixed • 149/150 fixed automaEcally • Only 23 false posiEves Applica%on Descrip%on LOC Bugs Ant Build tool 140,674 1 Groovy Dynamic language 161,487 9 JMeter Load tesEng tool 114,645 4 Log4J Logging framework 51,936 6 Lucene Text search engine 441,649 14 PDFBox PDF framework 108,796 10 Sling Web app. framework 202,171 6 Solr Search server 176,937 2 Struts Web app. framework 175,026 4 Tika Content extracEon 50,503 1 Tomcat Web server 295,223 4 Google Chrome Web browser 13,371,208 22 GCC Compiler 1,445,425 22 Mozilla Web browser 5,893,397 27 MySQL Database server 1,774,926 18