Bug prediction based on your code history
Download as PPTX, PDF1 like401 views
The document discusses bug prediction in software development using code history, highlighting the limitations of traditional practices like code reviews and automated testing. It describes a process of collecting data from commits to improve predictions of bug occurrences, utilizing machine learning tools such as Weka and implementing various classifiers. It concludes by noting the importance of analyzing commit history to reduce bugs and improve team practices.
Bug prediction based on your code history
- 1. Bug prediction based on your code history Alexey Tokar VP of Engineering @ WorldAPP
- 2. SDLC 2 Software Development Life Cycle PHASE 2 Design PHASE 1 Requirements analysis PHASE 3 Development PHASE 5 Maintenance PHASE 4 Testing
- 3. Zoom in to quality control ● static code analyzers find non conceptual issues ● automated tests cover predefined scenarios ● code review are aimed on sharing and controlling best practices and less than 10% of the discussions discover logical issues. ● and, finally, QA has no idea which parts of a system could be affected by a code change, neither do a programmer 3
- 4. 20 bugs in a production environment per week 4
- 5. Let's try to guess common patterns ● a tired engineer makes more mistakes ● the more an engineer knows about certain module the fewer bugs (s)he will produce ● small changes have fewer bugs than long listings ● some parts of the system are more complicated than another, so the risk of get a bug increases ● huge changes in a short period of time contains more bugs (done in a hurry) 5
- 6. What tools do we have across SDLC? 6 ● ticket types ● action history ● exact code changes ● author of modifications ● class complexity ● code metrics
- 7. Hypothesis If we know that certain commit has fixed a bug, than we know that a commit, when the changed lines were introduced, did contain a bug. 7 Author: John public int sum( int a, int b ) { return a + b; } C Author: Bob public int sum( int a, int b ) { return a * b; } BA public int sum( int a, int b ) { return a + b; }
- 8. Algorithm of metrics collection ● Export all tasks from Jira to inmemory dictionary ● For each commit run a backtrace to mark it as buggy, fixing or regular ● Collect all meaningful data about the commit: ○ Month of year, Day of week, Hour of day, Who, How many lines and files, Which classes and packages, Class complexity and amount of notices, How long a task is in progress ● Put a line with the data to Attribute-Relation File Format (ARFF) file 8
- 9. Getting educated. WEKA Waikato Environment for Knowledge Analysis - is a suite of machine learning software written in Java, developed at the University of Waikato, New Zealand. ● Parsers ● Classifiers ● Training/test splits 9
- 10. WEKA challenges ● Convert your data to corresponding vectors ● Choose proper data transformers ● Select and tweak desired Classifiers ● Run experiments and adjust your settings Good materials about WEKA for beginners: ● How to Run Your First Classifier in Weka ● Data mining with WEKA, Part 2. Classification and clustering ● Document Classification using WEKA 10
- 11. Decision Tree Ease of results interpretation Any data can be fed to the method Can work with scalars and intervals 11
- 12. Decision Tree 12 Changed less than 300 lines?Changed more than 50 lines? Author is Bob? Author is John? Has no bugs :) Has no bugs :) Is it Friday? Has no bugs :) Has a bug :( Has no bugs :) Has a bug :( ● John never has bugs! ● Everybody except John and Bob has bugs on Friday. ● Bob has bugs only if he changed more than 300 lines of code.
- 13. Decision Tree 13 The simplest method for building a tree is ID3 (Iterative Dichotomiser 3*). Build steps: ● Find an attribute with lowest entropy (or largest information gain) ● Split the data set by the found attribute ● Recursively build a tree for each of the subsets * fates of ID2 and ID1 are lost in history
- 14. Naive Bayes classifier ≈80% accuracy* Simple implementation Easy to understand 14
- 16. Naive Bayes classifier 16 30% of all commits with bugs were done by Bob P(Bob|bug) 10% of all commits without bugs were done by Bob P(Bob|~bug) 40% of all commits have bugs P(bug) 60% of all commits have no bugs P(~bug) What probability that next commit from Bob will have a bug? P(bug|Bob)
- 17. Support Vector Machine Better quality of results The model is based on relations in data Sounds fancy :) 17
- 19. Support Vector Machine 19 Kernel trick http://mechanoid.kiev.ua/ml-svm.html
- 21. Output results example (Bayes) Correctly Classified Instances 14381 77.4755 % Incorrectly Classified Instances 4181 22.5245 % Kappa statistic 0.3085 Mean absolute error 0.2637 Root mean squared error 0.3963 === Detailed Accuracy By Class === TP Rate FP Rate Precision Recall F-Measure ROC Area Class 0.856 0.544 0.861 0.856 0.858 0.761 false 0.456 0.144 0.444 0.456 0.45 0.761 true Weighted Avg. 0.775 0.463 0.777 0.775 0.776 0.761 === Confusion Matrix === a b <-- classified as 12670 2140 | a = false 2041 1711 | b = true 21
- 22. Output results example (RandomTree) form < 1 | Registration < 1 | | alexey.tokar@worldapp.com < 1 | | | tpl < 1 | | | | filters < 1 | | | | | frontend@worldapp.com < 1 | | | | | | middlejava@worldapp.com < 1 : false | | | | | | middlejava@worldapp.com >= 1 : true | | | | | frontend@worldapp.com >= 1 | | | | | | ObjectDesign < 1 : true | | | | | | ObjectDesign >= 1 : false | | | | filters >= 1 : false | | | tpl >= 1 : true | | alexey.tokar@worldapp.com >= 1 | | | bundle < 1 | | | | xmail < 1 | | | | | general < 1 | | | | | | dataimport < 1 | | | | | | | oracle < 1 : false | | | | | | | oracle >= 1 : true | | | | | | dataimport >= 1 : false | | | | | general >= 1 | | | | | | filesedited < 2 : false | | | | | | filesedited >= 2 : true | | | | xmail >= 1 : false | | | bundle >= 1 : true | Registration >= 1 : true 22
- 23. Integration example TL;DR: we use GitLab web-hooks and HipChat. * a long story about SDLC improvements with help of IM bot and a set of integrations will be available in a week at XPDays conference. 23
- 24. Summary ● we found that certain classes are too complex as almost every change will end up with a bug ● some of engineers shouldn't open some packages at all (or at least we should properly educate them) ● there are still many rooms for improvements (overlapping hiding commits, another meaningful features, more accurate code history, etc) ● It does not show you where the error exists. But you will be able to analyze a commit more carefully. ● It was fun! :) 24
- 25. Questions? Alexey@Tokar.net.ua VP of Engineering @ WorldAPP 25
Editor's Notes
- #5: Having all these activities we still get up to 20 bugs during a week in production environments reported by our customers
- #7: (it is worth to mention that accuracy is a must have for future improvements) crossreferences
- #8: Боб изменил функцию Джон прислал коммит с текстом fixed Отмечаем такой коммит как Fixing Находим измененные строки Делаем Blame и находим когда они были представлены Отмечаем такой коммит как Buggy
- #10: As it is a Java conference I should took a java based library for that ;)
- #15: classifier based on applying Bayes' theorem with strong (naive) independence assumptions between the features.
- #19: An SVM model is a representation of the examples as points in space, mapped so that the examples of the separate categories are divided by a clear gap that is as wide as possible. New examples are then mapped into that same space and predicted to belong to a category based on which side of the gap they fall.