Managing with Metrics Webinar Shaun Bradshaw and Philip lew
0 likes989 views
The document outlines a presentation by Shaun Bradshaw and Philip Lew on software testing metrics, focusing on the Rayleigh cumulative distribution's application in test execution progress and application stability tracking. It describes a manual test effort involving multiple SMEs and highlights execution challenges and successes over a 15-day period. The presentation emphasizes the importance of data requirements and metrics interpretation to ensure successful testing outcomes.
![What is Measurement?
• Measurement is the process by which numbers
or symbols are assigned to attributes of entities
in the real world in such a way as to characterize
them according to clearly defined rules. [1]
• Measurement is the empirical, objective
assignment of numbers, according to a rule
derived from a model or theory, to attributes of
objects or events with the intent of describing
them. [2]](https://image.slidesharecdn.com/managingwithmetricswebinar-shaunbradshaw-phillew-140218224638-phpapp01/85/Managing-with-Metrics-Webinar-Shaun-Bradshaw-and-Philip-lew-5-320.jpg)
![Rayleigh Cumulative Distribution
• In probability theory and statistics, the Rayleigh
distribution is a continuous probability distribution. As an
example of how it arises, the wind speed will have a
Rayleigh distribution if the components of the two-
dimensional wind velocity vector are uncorrelated and
normally distributed with equal variance. The distribution
is named after Lord Rayleigh. [1]
What is the Rayleigh Cumulative Distribution?](https://image.slidesharecdn.com/managingwithmetricswebinar-shaunbradshaw-phillew-140218224638-phpapp01/85/Managing-with-Metrics-Webinar-Shaun-Bradshaw-and-Philip-lew-6-320.jpg)
![Rayleigh Cumulative Distribution
• Test execution starts slowly as the team works
through configuration issues and major blocking
defects.
• Once the initial issues are resolved a larger
variety of tests can be executed, increasing
execution velocity.
• As testing nears release, there are fewer tests to
be executed and only a few defects remain
outstanding, leveling out the speed of execution.
Why does it have an S shape?[2]](https://image.slidesharecdn.com/managingwithmetricswebinar-shaunbradshaw-phillew-140218224638-phpapp01/85/Managing-with-Metrics-Webinar-Shaun-Bradshaw-and-Philip-lew-7-320.jpg)
Managing with Metrics Webinar Shaun Bradshaw and Philip lew
- 1. Shaun Bradshaw & Philip Lew Managing with Metrics The Saga of a Test Effort!
- 2. XBOSoft info • Founded in 2006 • Dedicated to software quality • Software QA consulting • Software testing services • Offices in Santa Clara,CA Beijing, Oslo and Amsterdam
- 3. About the Speakers Shaun Bradshaw VP Consulting Services Zenergy Technologies Philip Lew CEO and Founder XBOSoft
- 4. The Metric
- 5. What is Measurement? • Measurement is the process by which numbers or symbols are assigned to attributes of entities in the real world in such a way as to characterize them according to clearly defined rules. [1] • Measurement is the empirical, objective assignment of numbers, according to a rule derived from a model or theory, to attributes of objects or events with the intent of describing them. [2]
- 6. Rayleigh Cumulative Distribution • In probability theory and statistics, the Rayleigh distribution is a continuous probability distribution. As an example of how it arises, the wind speed will have a Rayleigh distribution if the components of the two- dimensional wind velocity vector are uncorrelated and normally distributed with equal variance. The distribution is named after Lord Rayleigh. [1] What is the Rayleigh Cumulative Distribution?
- 7. Rayleigh Cumulative Distribution • Test execution starts slowly as the team works through configuration issues and major blocking defects. • Once the initial issues are resolved a larger variety of tests can be executed, increasing execution velocity. • As testing nears release, there are fewer tests to be executed and only a few defects remain outstanding, leveling out the speed of execution. Why does it have an S shape?[2]
- 8. Data Requirements • The curve can be tracked for two main purposes with similar data requirements: – Test Execution Progress • Total tests to be executed • Total number of tests in a Passed state • Total number of days in the test effort – Application Stability Tracking • Total tests to be executed • Historical fail rates (to derive the anticipated number of failures) • Total number of failures • Total number of days in the test effort What data is required to generate the curve?
- 9. The Graph
- 10. Actual vs Theoretical By plotting the actual cumulative number of passed tests and comparing it to the theoretical curve we are able to identify potential issues and make adjustments to the effort ensuring testing is as successful as possible.
- 11. The Saga
- 12. • Team – No professional testers – 30-40 SMEs in the areas of: • Supply Chain Planning, Supply Chain Management, Manufacturing, Operations, Sales (divided by product category), Accounting, etc. • Test effort – Complete manual test execution of 370 E2E, 390 functional, and 600 process flow test cases in 15 days – Daily stand-up held to review metrics and adjust test execution strategy Interpreting the Curve
- 13. Day 1 Interpreting the Curve Good start, as expected. Since this is a regression test effort most environmental issues should be taken care of, as well as the few, if any major blocking issues
- 14. Pass rate at this point indicates a possible early completion. Process flow execution and passes have been high since the accounting group shifted resources to test execution. The best news is that the team is about 2 days ahead of schedule. Manager verifies with team that they are execut- ing high priority tests first. Interpreting the Curve Day 5
- 15. Execution has become anemic. Accounting is not completing process flow validations because of issues with taxes and a known problem with VAT. Next week is focused on defect correction and re-testing. Devel- opment indicates the custom manu- facturing issue will be corrected by the end of the week and VAT should be correct- ed as well. Interpreting the Curve Day 10
- 16. Although the team did not achieve 100% completion, fewer than 40 tests were outstanding. Furthermore, specific acceptance criteria were established prior to test execution and all criteria were met or exceeded. Interpreting the Curve Day 15
- 17. Questions
- 18. Follow: @XBOSoft Questions: services@xbosoft.com 408-350-0508 Follow: @Zenergytechnologies Questions: 877.375.7041