Finding Bugs, Fixing Bugs, Preventing Bugs - Exploiting Automated Tests to Increase Reliability
0 likes92 views
The document discusses advancements in automated testing for software development, emphasizing the importance of speed and quality in agile processes. It highlights the challenges of finding and fixing bugs, the role of mutation testing, and the integration of artificial intelligence for enhancing testing processes. The document also presents case studies and data on testing efficiency, discussing various test types and their effectiveness.
![© Serge Demeyer
Test Suite
13
TEST(FindLastTests, emptyVector) {
EXPECT_EQ(-1, findLast({}, 3));
}
TEST(FindLastTests, doubleOccurrence) {
EXPECT_EQ(3, findLast({1, 2, 42, 42, 63}, 42));
}
TEST(FindLastTests, noOccurrence) {
EXPECT_EQ(-1, findLast({1, 2, 42, 42, 63}, 99));
int findLast(std::vector<int> x, int y) {
if (x.size() == 0)
return -1;
for (int i = x.size() - 1; i >= 0; i--)
if (x[i] == y)
return i;
return -1;
}](https://image.slidesharecdn.com/effectsbarco2022slideshare-220331153822/85/Finding-Bugs-Fixing-Bugs-Preventing-Bugs-Exploiting-Automated-Tests-to-Increase-Reliability-13-320.jpg)
![© Serge Demeyer
Inject Mutant (Killed)
15
01 int findLast(std::vector<int> x, int y) {
02 if (x.size() == 0)
03 return -1;
04 for (int i = x.size() - 1; i < 0; i--)
05 if (x[i] == y)
06 return i;
07 return -1;
08 }
[ PASSED ] 2 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] FindLastTests.doubleOccurrence
Relational Operator Replacement (ROR)
“i >= 0” becomes “i < 0”](https://image.slidesharecdn.com/effectsbarco2022slideshare-220331153822/85/Finding-Bugs-Fixing-Bugs-Preventing-Bugs-Exploiting-Automated-Tests-to-Increase-Reliability-15-320.jpg)
![© Serge Demeyer
Inject Mutant (Survived - Live)
16
01 int findLast(std::vector<int> x, int y) {
02 if (x.size() == 0)
03 return -1;
04 for (int i = x.size() - 1; i > 0; i--)
05 if (x[i] == y)
06 return i;
07 return -1;
08 }
[==========] 3 tests from 1 test suite ran. (0 ms total)
[ PASSED ] 3 tests.
Relational Operator Replacement (ROR)
“i >= 0” becomes “i > 0”](https://image.slidesharecdn.com/effectsbarco2022slideshare-220331153822/85/Finding-Bugs-Fixing-Bugs-Preventing-Bugs-Exploiting-Automated-Tests-to-Increase-Reliability-16-320.jpg)
![© Serge Demeyer
TEST(FindLastTests, occurrenceOnBoundary) {
EXPECT_EQ(0, findLast({1, 2, 42, 42, 63}, 1));
}
[==========] 4 tests from 1 test suite ran. (0 ms total)
[ PASSED ] 3 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] FindLastTests.occurrenceOnBoundary
Strengthening the Test Suite
17
01 int findLast(std::vector<int> x, int y) {
02 if (x.size() == 0)
03 return -1;
04 for (int i = x.size() - 1; i > 0; i--)
05 if (x[i] == y)
06 return i;
07 return -1;
08 }](https://image.slidesharecdn.com/effectsbarco2022slideshare-220331153822/85/Finding-Bugs-Fixing-Bugs-Preventing-Bugs-Exploiting-Automated-Tests-to-Increase-Reliability-17-320.jpg)
![© Serge Demeyer
Example - testDeposit_ampli
fi
ed (1/2)
38
1 def testDeposit_amplified ( self ) :
2 self.b.set_owner(’Iwena Kroka’)
3 self.b.deposit(10)
4 self.assertEqual(self.b.
5 get_transactions(), [10])
6 self.assertFalse (self .b. is_empty () )
7 self.assertEqual(self.b.owner, ’Iwena Kroka’)
8 self.assertEqual(self.b.get_balance(), 10)
…
Assertion
Am
plification](https://image.slidesharecdn.com/effectsbarco2022slideshare-220331153822/85/Finding-Bugs-Fixing-Bugs-Preventing-Bugs-Exploiting-Automated-Tests-to-Increase-Reliability-38-320.jpg)
![© Serge Demeyer
Example - testDeposit_ampli
fi
ed (2/2)
39
1 def testDeposit_amplified ( self ) :
2 self.b.set_owner(’Iwena Kroka’)
3 self.b.deposit(10)
4 self.assertEqual(self.b.
5 get_transactions(), [10])
6 self.assertFalse (self .b. is_empty () )
7 self.assertEqual(self.b.owner, ’Iwena Kroka’)
8 self.assertEqual(self.b.get_balance(), 10)
9 with self.assertRaises(Exception):
10 self .b. deposit(−56313)
11 self.b.deposit(100)
12 self.b.set_owner(’Guido van Rossum’)
13 self.assertEqual(self.b.
14 get_transactions(), [10, 100])
…
Input Am
plification](https://image.slidesharecdn.com/effectsbarco2022slideshare-220331153822/85/Finding-Bugs-Fixing-Bugs-Preventing-Bugs-Exploiting-Automated-Tests-to-Increase-Reliability-39-320.jpg)
Finding Bugs, Fixing Bugs, Preventing Bugs - Exploiting Automated Tests to Increase Reliability
- 1. Finding Bugs, Fixing Bugs, Preventing Bugs Exploiting Automated Tests to Increase Reliability March 2022 — © Prof. Serge Demeyer
- 4. Ericsson, Bombardier, Saab, System Veri fi cation, Empear, Verifyter, KTH, MDH, RISE Comiq, E fi Code, Ponsse, Siili, Qentinel, Symbio, Uni.Oulu, VTT Axini, Testwerk, TNO, Open Uni. AKKA, Expleo, EKS, FFT, Fraunhofer, IFAK, OFFIS, Parasoft Alerion, Prodevelop,, Uni.Mandragon Kuveyt Bank, Saha BT The TESTOMAT project will allow software teams to increase the development speed without sacri fi cing quality. To achieve this goal, the project will advance the state-of-the-art in test automation for software teams moving towards a more agile development process.
- 6. Six decades into the computer revolution, four decades since the invention of the microprocessor, and two decades into the rise of the modern Internet, all of the technology required to transform industries through software fi nally works and can be widely delivered at global scale.
- 9. Finding Bugs, Fixing Bugs, Preventing Bugs Exploiting Automated Tests to Increase Reliability March 2022 — © Prof. Serge Demeyer
- 10. © Serge Demeyer Continuous Integration Pipeline 10 <<Breaking the Build>> version control build developer tests deploy scenario tests deploy to production measure & validate
- 11. Finding Bugs, Fixing Bugs, Preventing Bugs Exploiting Automated Tests to Increase Reliability March 2022 — © Prof. Serge Demeyer
- 12. c++ java
- 13. © Serge Demeyer Test Suite 13 TEST(FindLastTests, emptyVector) { EXPECT_EQ(-1, findLast({}, 3)); } TEST(FindLastTests, doubleOccurrence) { EXPECT_EQ(3, findLast({1, 2, 42, 42, 63}, 42)); } TEST(FindLastTests, noOccurrence) { EXPECT_EQ(-1, findLast({1, 2, 42, 42, 63}, 99)); int findLast(std::vector<int> x, int y) { if (x.size() == 0) return -1; for (int i = x.size() - 1; i >= 0; i--) if (x[i] == y) return i; return -1; }
- 14. 100% line coverage 100% statement coverage 100% branch coverage all tests passed
- 15. © Serge Demeyer Inject Mutant (Killed) 15 01 int findLast(std::vector<int> x, int y) { 02 if (x.size() == 0) 03 return -1; 04 for (int i = x.size() - 1; i < 0; i--) 05 if (x[i] == y) 06 return i; 07 return -1; 08 } [ PASSED ] 2 tests. [ FAILED ] 1 test, listed below: [ FAILED ] FindLastTests.doubleOccurrence Relational Operator Replacement (ROR) “i >= 0” becomes “i < 0”
- 16. © Serge Demeyer Inject Mutant (Survived - Live) 16 01 int findLast(std::vector<int> x, int y) { 02 if (x.size() == 0) 03 return -1; 04 for (int i = x.size() - 1; i > 0; i--) 05 if (x[i] == y) 06 return i; 07 return -1; 08 } [==========] 3 tests from 1 test suite ran. (0 ms total) [ PASSED ] 3 tests. Relational Operator Replacement (ROR) “i >= 0” becomes “i > 0”
- 17. © Serge Demeyer TEST(FindLastTests, occurrenceOnBoundary) { EXPECT_EQ(0, findLast({1, 2, 42, 42, 63}, 1)); } [==========] 4 tests from 1 test suite ran. (0 ms total) [ PASSED ] 3 tests. [ FAILED ] 1 test, listed below: [ FAILED ] FindLastTests.occurrenceOnBoundary Strengthening the Test Suite 17 01 int findLast(std::vector<int> x, int y) { 02 if (x.size() == 0) 03 return -1; 04 for (int i = x.size() - 1; i > 0; i--) 05 if (x[i] == y) 06 return i; 07 return -1; 08 }
- 18. © Serge Demeyer Industrial Case Study 18 • 83K lines of code • Complicated structure • Lots of legacy code • Lots of black-box tests Ali Parsai, Serge Demeyer; “Comparing Mutation Coverage Against Branch Coverage in an Industrial Setting”. Software Tools for Technology Transfer
- 19. Unit tests only ! Segmentation Percentage 0 20 40 60 80 100 Mutation Coverage Branch Coverage
- 20. CI D evelop Bu ild Test W ay too slow We witnessed 48 hours of mutation testing time on a test suite comprising 272 unit tests and 5,258 lines of test code for testing a project with 48,873 lines of production code. Sten Vercammen, Serge Demeyer, Markus Borg, and Sigrid Eldh; “Speeding up Mutation Testing via the Cloud: Lessons Learned for Further Optimisations”. Proceedings ESEM 2018
- 21. Master 1) Initial test Build 2) ∀ files to mutate: queue file names 3a) Generate mutants 4a) Execute mutants 3b) Store mutants 4b) Store results 3c) Queue mutant references 5) Process results
- 22. 0 1h 2h 3h LittleDarwin 1 worker 2 workers 4 workers 8 workers 16 workers 0 12h 1d 1d 12h 2d 2d 12h LittleDarwin 1 worker 2 workers 4 workers 8 workers 16 workers
- 24. © Serge Demeyer DO-178C = Avionics Safety Standard 24 Mutation Testing = Actionable !
- 25. Finding Bugs, Fixing Bugs, Preventing Bugs Exploiting Automated Tests to Increase Reliability March 2022 — © Prof. Serge Demeyer
- 26. © Serge Demeyer 4 Quadrants 26 Supporting Team Functional Tests Examples Story Tests Prototypes Simulation Exploratory Testing Scenarios Usability Testing Acceptance Testing Alpha / Beta Unit Tests Integration Tests Performance Testing Load Testing Security Testing “ility” Testing Technology Facing Business Facing Critique Product Automated & Manual Manual Autom ated Tools
- 27. © Serge Demeyer Flipping the V 27 70% 20% 10% 10% 20% 70% Acceptance Tests (GUI Tests) System Tests Integration Tests Component Tests Unit Tests Manual Automated
- 28. © Serge Demeyer Flipping the V? 28
- 29. © Serge Demeyer Flakey Tests 29 version control build developer tests deploy scenario tests deploy to production measure & validate Non-determistic Result
- 30. © Serge Demeyer Flakey Tests: Taxonomy 30 CAUSES Avoidable Async Wait Concurrency I/O Memory Arithmetic Randomness Collections Inevitable Time Test Order External Dependencies
- 31. © Serge Demeyer Flakey Tests: Management 31 Zero Tolerance Manage
- 32. Finding Bugs, Fixing Bugs, Preventing Bugs Exploiting Automated Tests to Increase Reliability March 2022 — © Prof. Serge Demeyer
- 33. Advance the state-of-the-art in autonomous (= zero-touch) testing for DevOps software teams. Arti fi cial Intelligence Inside
- 34. © Serge Demeyer Testing 34 Program Under Test Input Expected output Software Testing is the process of executing a program or system with the intent of finding errors. (Myers, Glenford J., The art of software testing. Wiley, 1979
- 35. © Serge Demeyer Coverage 35 Program Under Test Input Expected output (mutation) coverage
- 36. © Serge Demeyer Test Ampli fi cation 36 Program Under Test Input Expected output (mutation) coverage +coverage Extra Input Extra Input +Extra Input +Extra output Arti fi cial Intelligence Inside
- 37. © Serge Demeyer Example - testDeposit 37 1 def testDeposit ( self ) : 2 self.b.set_owner(’Iwena Kroka’) 3 self.b.deposit(10) 4 self.assertEqual(self.b.get_balance(), 10) 5 self.b.deposit(100) 6 self.b.deposit(100) 7 self.assertEqual(self.b.get_balance() , 210) I n p u t Expected output
- 38. © Serge Demeyer Example - testDeposit_ampli fi ed (1/2) 38 1 def testDeposit_amplified ( self ) : 2 self.b.set_owner(’Iwena Kroka’) 3 self.b.deposit(10) 4 self.assertEqual(self.b. 5 get_transactions(), [10]) 6 self.assertFalse (self .b. is_empty () ) 7 self.assertEqual(self.b.owner, ’Iwena Kroka’) 8 self.assertEqual(self.b.get_balance(), 10) … Assertion Am plification
- 39. © Serge Demeyer Example - testDeposit_ampli fi ed (2/2) 39 1 def testDeposit_amplified ( self ) : 2 self.b.set_owner(’Iwena Kroka’) 3 self.b.deposit(10) 4 self.assertEqual(self.b. 5 get_transactions(), [10]) 6 self.assertFalse (self .b. is_empty () ) 7 self.assertEqual(self.b.owner, ’Iwena Kroka’) 8 self.assertEqual(self.b.get_balance(), 10) 9 with self.assertRaises(Exception): 10 self .b. deposit(−56313) 11 self.b.deposit(100) 12 self.b.set_owner(’Guido van Rossum’) 13 self.assertEqual(self.b. 14 get_transactions(), [10, 100]) … Input Am plification
- 40. 11 pull requests 8 merged 3 pending
- 41. © Serge Demeyer Research Agenda 41 Program Under Test (mutation) coverage +coverage Arti fi cial Intelligence Inside (Intention Revealing) Test Names Explainable Test Cases Visualisation (Dashboards) Work fl ow (Github Actions)
- 42. Finding Bugs, Fixing Bugs, Preventing Bugs Exploiting Automated Tests to Increase Reliability March 2022 — © Prof. Serge Demeyer
- 43. Description text Mining Stack Traces Link to source code Product/Component Speci fi c vocabulary Suggestions?
- 44. Question Cases Precision Recall Who should fi x this bug? Eclipse, Firefox, gcc eclipse: 57% fi refox: 64% gcc: 6% — How long will it take to fi x this bug? (*) JBoss depends on the component many similar reports: off by one hour few similar reports: off by 7 hours What is the severity of this bug? (**) Mozilla, Eclipse, Gnome mozilla, eclipse:67% - 73% gnome: 75%-82% mozilla, eclipse:50% - 75% gnome: 68%-84% (*) In CSMR2012 Proceedings Who should fi x this bug? Eclipse, Firefox, gcc eclipse: 57% fi refox: 64% gcc: 6% — Irrelevant for Practitioners (**) In CSMR2011; MSR 2010 Proceedings Arti fi cial Intelligence Inside Internal vs. External Bug Reports
- 45. © Serge Demeyer Story Points (Planning Poker) 45 1/2 1 2 3 5 8 13 20 40 100 ♾ Public Domain
- 46. Human MMRE: 0.48 (*) Mean Magnitude of Relative Error Learning Curve
- 47. © Serge Demeyer Arti fi cial Intelligence Inside “in vivo” Validation 47 Explainable!
- 48. Finding Bugs, Fixing Bugs, Preventing Bugs Exploiting Automated Tests to Increase Reliability March 2022 — © Prof. Serge Demeyer
- 49. © Serge Demeyer Q&A support 49
- 51. © Serge Demeyer Hype Cycle 51 Hype Cycle © Gartner Visibility Maturity Technology Trigger Peak of Inflated Expectations Trough of Disillusionment Slope of Enlightenment Plateau of Productivity
- 52. © Serge Demeyer The Future ? 52 Personal Opinion Peak of Inflated Expectations Hype Cycle © Gartner Visibility Maturity Technology Trigger Trough of Disillusionment Slope of Enlightenment Plateau of Productivity IBM Microsoft Google FaceBook … … for an experiment nearby