Expanding our Testing Horizons
- 1. Expanding our Testing Horizons Mark Micallef PEST Research Lab, Malta Monday, July 28, 2014
- 8. Two Worlds • Understanding • Clean Solutions • Complete • Perfect • Value • Pragmatic • Working
- 10. Idea: Catching problems earlier makes them easier to fix
- 14. The Problem Static analysis tools can identify thousands of alerts This leads to cognitive overload… … which in turn leads to… … the technique being abandoned Concept: Actionable Alert Identification Techniques (AAIT)
- 16. Some discoveries 0 1 2 3 4 5 Priority of alert Size/Length of the method Size/Length of the class Age of alert Date of last modification Code churn Complexity of a method Nesting of a line of code Code (line) coverage NumberofInterviewees Very useful Somewhat useful Slightly useful Irrelevant Missing
- 17. A context-specific AAIT Expert System Alerts Context Filtered and Ranked Alerts
- 18. Idea: We should be aware of the effectiveness of our test suites
- 20. Statement coverage can be misleading public int multiply(int x, int y){ return (x y); } @Test public void testMultiply(){ assertEquals(5, multiply(5, 1)); } */ Passes x
- 21. Mutation Testing
- 22. Mutation Testing Test Suite Mutant 1 Mutant n Program
- 23. Problems with Mutation Testing Generating mutants is expensive Executing tests is expensive Susceptible to equivalent mutants Manual investigation of unkilled mutants
- 24. Applying context Version n δ δ δ δ Version n+1 δ δ Version n+1 changes δ δ δ δ δ δ
- 25. The results Code Churn Affected LOC Unchanged code Decrease in mutants Decrease in execution time Low 12 99.8% 91% 91% Medium 60 98.8% 62% 88% High 720 85.6% 46% 89%
- 26. Insight: Testing will never provide guarantees that bugs do not exists
- 27. Testing Activities Test Planning Test Case Design Execution Reporting & Management
- 28. Testing Activities Test Planning Test Case Design Execution Reporting & Management Runtime Testing
- 29. Testing (Multiple Paths) Runtime Testing (Single user-generated Path) Runtime Testing
- 30. Problems with Runtime Testing Defining mathematical properties may not come naturally to people Performance Overheads How do you fit this into your development process?
- 31. Applying Context Given I am a bank teller processing a transaction And account 67 has $100 and account 113 has $50 When I transfer $50 from account 67 to account 113 Then account 67 will have $50 And account 113 will have $100 And this should happen within 3 seconds And the transfer should be logged And a receipt should be printed And the respective clients should be notified Relevant to feature Cross-Cutting Criteria
- 32. Applying Context Cross-Cutting Criteria Feature 1 Feature 2 Feature 3
- 33. What can I do?
- 34. You can… Periodically scan academic journals for new ideas Try Google Scholar, set up alerts for your favourite topics Sell the benefits of research to yourself and colleagues Actively seek out collaborations with research groups, local or otherwise Consider pursuing research for academic credit yourself or sponsor and employee to do so
- 35. Some collaboration models Provide a case study Sponsor an intern Hand off a problem to a research group Contribute to a research trust