Are You Making These 7 'Testing Metric' Mistakes? Webinar - Mark Bentsen, Philip Lew

Editor's Notes

• #11: Reporting upward Goal is to build confidence that their involvement is not needed unless requested Content / details varies for audience (Team, Managers, Directors, Officers) Concise, accurate, confidence-building Start with less, but make sure you have asked the right questions and have answers Flows always help- high level, block flows with associated test cases
• #12: Communication Heuristics – Does not always happen in email Always needs a headline. Would you read a newspaper or magazine article without one? What else do they use in their format? After meetings – big game hunting In an elevator Take the elevator challenge Ground to top floor/to lowest floor Kung Fu Reporting Take an opportunity (they call you) and you turn it When you “do” send email, make sure it is relevant. When are the meetings where people need your info. When do PM’s begin preparing their reports? Your report needs to be in their outlook before they start that process
• #14: Flash In The Pan. Symptom: Too ambitious in the beginning. Principle: Slow is fast. Solution: Make a good decision up front: Then: Steadily report meaningful information Double is trouble Metrics on the same thing in multiple ways Be sure to be consistent
• #15: Heavy Lifting – Choosing to work hard instead of smart. Symptom: Overly manual processes required to produce. Principle: Maintainability is a key quality attribute and a success criteria for process success. Solution – Build in telemetry Then: Automate, or automation assist.
• #16: Message In A Bottle – communicating in a vacuum, throwing the bottle into the ether and expecting the best Symptom: No one responds to status reports from testing. Principle: Communication is what the listener (recipient) does. Solution – Know your audience. Who they are and what they care about. Sample informal review of internal customers to your testing organization What do you need & want from QA? How well are we meeting your needs? What is your perception of the QA team? (tell me the truth…) Then: Answer the questions they have and address their concerns.
• #17: Message In A Bottle – communicating in a vacuum, throwing the bottle into the ether and expecting the best Symptom: No one responds to status reports from testing. Principle: Communication is what the listener (recipient) does. Solution – Know your audience. Who they are and what they care about. Sample informal review of internal customers to your testing organization What do you need & want from QA? How well are we meeting your needs? What is your perception of the QA team? (tell me the truth…) Then: Answer the questions they have and address their concerns.
• #18: House on Fire. – The house is always on fire Symptom: Ongoing state of emergency is a symptom of poor reporting and insight into QA processes through reporting Principle: Reporting should be part of every testing process Solution: Validate you have meaningful metrics for each stage of your testing processes leading into a milestone. Are you on Track? What to adjust? Then: Remove the mystery from what is taking place
• #19: House on Fire. – The house is always on fire Symptom: Ongoing state of emergency is a symptom of poor reporting and insight into QA processes through reporting Principle: Reporting should be part of every testing process Solution: Validate you have meaningful metrics for each stage of your testing processes leading into a milestone. Are you on Track? What to adjust? Then: Remove the mystery from what is taking place
• #20: Mine Goes To Eleven – The opposite of simple Symptom: Exaggerated, Hollow, Pretentious. No one understands the metrics or the implications; this includes the person sending it out. Principle – Keep it simple. Gate guard example Solution: Choose to be pragmatic over trying to be impressive. Then: Be consistent. Be crystal clear in your narratives. Bonus – Measure surprises Good Bad Don’t let your reports surprise any of the recipients – prewire Forgetting who they are for – The “Order” of understanding You Then others, “who” can only understand the metrics if you do first Can you interpret what this means to your plan? Assumption: You have a plan.
• #21: Mine Goes To Eleven – The opposite of simple Symptom: Exaggerated, Hollow, Pretentious. No one understands the metrics or the implications; this includes the person sending it out. Principle – Keep it simple. Gate guard example Solution: Choose to be pragmatic over trying to be impressive. Then: Be consistent. Be crystal clear in your narratives. Bonus – Measure surprises Good Bad Don’t let your reports surprise any of the recipients – prewire Forgetting who they are for – The “Order” of understanding You Then others, “who” can only understand the metrics if you do first Can you interpret what this means to your plan? Assumption: You have a plan.
• #22: Are you providing numbers, but not answering relevant key questions for your senior stakeholders?   The traditional questions Executives ask, “Are they outdated?” How many test cases did you run? How many are automated? How many bugs did you find? Defect metrics / Trends?   You learned that QA probably can’t cover everything… What is important for you? Performance? Core functions? What didn’t QA test? What are the limitations? What are you afraid of? (don’t need data give me a gut feel) Would more time help or not? Appropriate Trends/Metrics   Simple Data points followed up by complex appendix…   When reporting: Start with a simple Yes or NO (on a release vote) Gut check, what am I worried about… Go a bit deeper with feature level data (Check, Check + etc...) Schedule compression number QA typically never gets the X days requested to test Publish a compression number to indicate risk We only had 10/15 days so 0.67 compression (risk is high) If you had calculated you need a full tank of gas to make it to Houston, would you go with 2/3? Go a bit deeper with regression coverage maps NO Test cases run, or defect metrics etc…   Example: Yes , Go Release Areas for concern XYZ Because we got it late and configuration nightmare to test Schedule Compression = 0.67 (we got crunched badly) Test Case point total = 892/1000 (we got close but not all the way) New feature Coverage Map 2 areas yellow rest 6 Green   Appendix: Test Cases run, defects trends, LOKI, Intermittent defect list etc., etc.… Mitigations for all of the above when possible Looks like a 1 page go/no go/ with a large appendix Think ESPN Box score, but with full stats later down in the article Also: – Wrong context of report for the testing Example – Functional API testing vs. Functional testing executed via Quality Center
• #23: Are you providing numbers, but not answering relevant key questions for your senior stakeholders?   “Are they outdated?” The traditional questions Executives ask: How many test cases did you run? How many are automated? How many bugs did you find? Defect metrics / Trends?   You learned that QA can’t cover everything… What is important for you? Performance? Features and Functionality (F&F), especially critical functions? What didn’t QA test? What are the limitations? What are you afraid of? (don’t need data give me your thoughts based on intuition and expertise) Would more time help or not? Appropriate Trends/Metrics   Simple Data points followed up by comprehensive appendix…   When reporting: Start with a simple Yes or NO (on a release vote) Reality check, what am I worried about… Go a bit deeper with feature level data (Check, Check + etc...) Schedule compression number QA typically never gets the X days requested to test Publish a compression number to indicate risk We only had 10/15 days so 0.67 compression (risk is high) If you had calculated you need a full tank of gas to make it to Houston, would you go with 2/3? Go a bit deeper with regression coverage maps NO Test cases run, or defect metrics etc…   Example: Yes , Go Release Areas for concern XYZ Because we got it late and configuration nightmare to test Schedule Compression = 0.67 (we got crunched badly) Test Case point total = 892/1000 (we got close but not all the way) New feature Coverage Map (2) areas yellow rest (6) Green   Appendix: Test Cases run, defects trends, Intermittent defect list etc., etc.… Mitigations for all of the above when possible Looks like a 1 page go/no go/ with a large appendix Think Horse race grid, but with full stats provided below or in this case, later in the status Also: – Wrong context of report for the testing Example – Functional API testing vs. Functional testing executed via Quality Center
• #24: Rearview Mirror Driver – Constant focus on the past without forecasting the future. Symptom: You are spending all your time trying to find defects instead of time allocated to activities that would prevent them. Principle: An ounce of prevention is worth a pound of cure. Solution: Predict what your metrics will be in the next two weeks. Document it. Then: Compare reality with your predictions. Adjust as appropriate until you can accurately forecast.   SDLC Quality & Root Causes Are you capturing root causes or excuses? ARGO thirteen Definitions Accountability Process Make it actionable Predict outcomes and measure results Wash, rinse, and repeat
• #25: Rearview Mirror Driver – Constant focus on the past without forecasting the future. Symptom: You are spending all your time trying to find defects instead of time allocated to activities that would prevent them. Principle: An ounce of prevention is worth a pound of cure. Solution: Predict what your metrics will be in the next two weeks. Document it. Then: Compare reality with your predictions. Adjust as appropriate until you can accurately forecast.   SDLC Quality & Root Causes Are you capturing root causes or excuses? ARGO thirteen Definitions Accountability Process Make it actionable Predict outcomes and measure results Wash, rinse, and repeat
• #26: Guidance Regarding What To Do About It
• #27: The Test Management Trifecta What have you completed? What did you learn? Defects Risks What testing remains?
• #28: Tactical vs. Strategic Tactical – Standardize across teams Strategic – Align to KPIs, YoY performance, Enterprise Quality Roadmap 97/97/250 Find 97% of the defects Fix 97% of those prior to release Drive the cost of a function point of development to $250 System health assessment. Need to know the definition of “healthy” in your context. GA Readiness Version Release (VR) Exit Criteria
• #29: And Then (and then, and then) BLUF – know your bottom line and lead with that. Think in terms of triple constraint. Schedule – We need an additional three days to complete planned testing. We lost five days to configuration management issues, but are working more efficiently as a team, so I’m confident this amount of time will be sufficient. Scope – We will now need to define additional test scenarios based on the change request getting approved Resources – We need two people from the product team to assist us in testing the new logic for the business rules Quality – We would like an additional three days to test functionality that had higher than anticipated defects found in regression. Doug’s advice – Don’t have the cost of the project on the final slide of the deck. Start with it. This is going to cost a million dollars, and here’s why…
• #30: Less is more. "I would have written you a shorter letter if I had more time." - Blaise Pascal There are three habits that keep most people from being succinct: The habit of over explaining The habit of underpreparing The habit to completely miss the point