Agile Testing Process Analytics: From Data to Insightful Information
- 1. T11 Metrics 10/6/16 11:15 Agile Testing Process Analytics: From Data to Insightful Information Presented by: Jonathan Alexander QA Symphony Brought to you by: 350 Corporate Way, Suite 400, Orange Park, FL 32073 888-‐-‐-‐268-‐-‐-‐8770 ·∙·∙ 904-‐-‐-‐278-‐-‐-‐0524 -‐ info@techwell.com -‐ http://www.starwest.techwell.com/
- 2. Jonathan Alexander As CTO at QASymphony, Jonathan Alexander leads the research and development of the qTest platform, including the development of new product offerings. With extensive experience in using analytics to understand and improve software teams and quality, Jonathan has written articles on this subject and authored Codermetrics: Analytics for Improving Software Teams. Previously, Jonathan was CTO at cloud-‐based business communications company Vonage Business Solutions. He has led development and testing teams at three start-‐upsÛÓVocalocity, vmSight, and RadnetÛÓand served in post-‐acquisition executive roles at Liquidware Labs and Epiphany. Jonathan began his career writing software for author/director Michael Crichton.
- 3. Agile Testing Analytics STARWEST 2016
- 4. Presenter Jonathan Alexander CTO of QASymphony
- 5. SDLC QASymphony - A Robust Testing Platform Monitor PPM Require- ments Build Test Defects Deploy Automation
- 6. Our Solutions For Test Case Management qTest is a powerful, elegant, scalable test management solution that works with small teams and allows large enterprises to coordinate and track hundreds of projects across many locations. For Unscripted Testing qTest eXplorer is a ground-breaking test documentation tool that supports exploratory and unscripted testing AND saves time when performing traditional manual testing. For Enterprise Reporting Insights 2.0 gives the testing team a self-service business intelligence tool to consolidate, manage and analyze all your testing data.
- 8. In This Session Analytics - What and Why Quality Analytics Coverage and Risk Analytics Velocity Analytics Test Case Optimization How to Get Started
- 10. Analytics - Data Sources Many potential data sources Need to create links: dev tickets -> code check-ins, test cases, defects, support cases
- 11. The Goal: Leverage Data for Improvement ● Use analytics to improve: ○ Test coverage ○ Forecasting completion dates ○ Efficiency and most effective use of resources ○ Test case quality ○ Individual performance ● Think of analytics as an objective input, not a magic bullet
- 12. Quality Analytics ● Core: ○ Test result %s by project ■ separate out latest runs ○ Defect priority and status %s ● Extra: ○ Test results by day or week ○ Test result %s by requirement ○ Severe defect counts ○ Defect status/priority crosstab ○ Defects per test run ○ Defect leakage (found by users) ● Tips: ○ Use color-coding to identify potential issues ○ Put manual and automated results side-by-side
- 13. Coverage Analytics ● Core: ○ Test cases by requirement ■ Latest run results ■ Breakdown by type ○ Defects by requirement ● Extra: ○ Test case complexity ○ Testing time per test case ● Tip: ○ Use data visualization to spot risks
- 14. Velocity Analytics ● Core: ○ Requirements inflow rate ○ Test case creation rate ○ Test run rate (cases & steps) ○ % tests complete and blocked ○ Defects opened and closed ● Extra: ○ Avg. test case complexity ○ Testing time spent and remaining ● Tip: ○ Breakdown analytics by tester
- 15. Test Case Optimization ● Start to think of test cases like source code ● Track both manual and automated (integration) test cases ● Track analytics that will help optimize your test case library ○ Days since last run ■ Archive test cases that are not used anymore ○ Flapping (# of times consecutive runs have different results) ■ Examine these tests and code/functional areas, might indicate need to refactor one or the other ○ Percentile complexity (steps and time spent per test) ■ Refactor highly complex tests for greater efficiency and more pinpoint understanding of results ○ Cumulative execution time ■ Automate the manual tests that are taking the most time ■ Refactor automated tests that are running the longest
- 16. How To Do This Yourself ● Setup a Test Analytics Reporting Server ○ Use an open source or 3rd party BI tool (such as qTest Insights) ○ Keep it simple ● Identify team members that will Work on Test Analytics ○ Depending on tool(s) may need technical and “analyst” ○ Commit to a certain # of hours per week or per month ● Start with Requirements, Test Results, and Defects data ○ For most companies data size will be very manageable ○ Don’t tackle big data problems (partition data if necessary) ● Start with Quality analytics, then add Coverage, then Velocity ○ Focus on weekly project reports ● Add More Detail and More Data Sources Over Time
- 17. Thank You for Listening! Questions?