Metrics: Tell A Story Not A Number by Mike Lyles
Download as PPTX, PDF2 likes304 views
The document provides advice and best practices for using metrics and measurements to tell an effective story, rather than just providing numbers, through discussions and exercises on defining good metrics, identifying issues with traditional metrics, and how to invoke change in organizations stuck relying on bad metrics. It cautions that measurements should have a clear purpose and relationship to what is being measured to avoid misinterpretation and unintended behaviors. Experts recommend focusing on the underlying questions and story, involving stakeholders, and refusing to condone misleading numbers rather than just counting for the sake of counting.
Metrics: Tell A Story Not A Number by Mike Lyles
- 1. Metrics: Tell A Story, Not A Number Mike Lyles www.MikeWLyles.com
- 2. About Me 2 QA Director with over 20 years in IT: • Development • Project Management Office • Development Manager • Testing • Functional Testing • Test Environments • Software Configuration Management • Test Data Management • Performance Testing • Test Automation • Service Virtualization • Building Testing Organizations • Defining Processes, Methodologies, Measurement
- 3. Once Upon A Time…
- 4. Workshop Takeaways • How to evaluate your current measurement program for gaps and areas of improvement • Group discussion on what others are doing • Inputs from industry leaders on metrics and measurement • Questions and Answers that you and your stakeholders need to ask to connect metrics with the most important objectives • Hands on exercises to coach attendees on the RIGHT “story” vs. the “numbers” • Developing a measurement program that builds the story and communicates status clearly
- 5. High Level Agenda • Your Story • Measuring • Numbers • Me • Storytelling • Health vs. Metrics • Metrics vs. Behavior • Misleading Metrics • Asking Questions • Correlations • Measurement vs. Metrics • Good, Bad, Misleading Metrics • Expert Advice • Reporting Examples
- 6. Exercise 1: What is Your Story?
- 7. What Is The Story? • Does your customer have faith in you? • Do you KNOW what to report? • Are you being asked to do something you know is not right? • Your customers are HUNGRY for information • They are waiting on you to do the right thing Build the right metrics and they will listen to your story! 7
- 8. Exercise 2: Let’s Measure Something...
- 9. “Not everything that can be counted counts, and not everything that counts can be counted.” Albert Einstein Don’t measure something just because you can. ScottAdams,http://www.dilbert.com 9
- 10. How About These Numbers? 1 3 1 4 2 3 19 1 21 150+ Conference Workshop Day Workshops Conference Days Keynotes Social Event on Day Two 10
- 12. I Am Not Just A Number! 8,29,71 2, 14, 9 89, 93 $20,000 1994, 19.5 6-1 Green, Brown 248 12
- 13. Sometimes Numbers Make Sense… • 4-8-15-16-23-42 • 26/01 • 07/03 • 01/11 • 365 • 24 • 7 • 292 • Bones in body • 7 • Continents • 196 • Countries • 8 • Planets • 1 • 1 13
- 14. Storytelling Mister Rogers carried a note from a social worker that said… “Frankly there isn’t anyone you couldn’t love once you’ve heard their story”. 14
- 15. Exercise 4: Let’s Hear Your Story?
- 16. Let Me Tell You A Joke!
- 17. The Art of Storytelling “Make Me Care” “Making a promise this story will lead somewhere that is worth your time” “The audience wants to work for their meal. They just don’t want to know they are doing it” http://www.ted.com/talks/andrew_stanton_the_clues_to_a_great_story?language=en 17
- 18. The Art of Storytelling “Give them 2+2. Don’t give them 4”. “Wonder can’t be artificially invoked. Have them surrender to wonder” 18
- 19. Your Health: Metrics for You… You monitor your health daily • Watching your calories • Measuring your weight • Controlling your intake • Taking your vitamins • Planning your exercise • Planning your sleep 19
- 20. Your Health: Metrics for the Doctor… Your doctor checks you yearly: • Looks at overall health • Less concerned about what you did day by day • Can determine if you were healthy through the year without knowing all the daily details 20
- 21. YOU verses the Doctor “Individual Health” • Daily reports • Defect reviews • Adjustments to test plan “Doctor Check-Up” • Executive reporting • Overall project status • Risks / Mitigations 21
- 23. Everyone has a part of the big picture 23
- 24. 24 Everyone has a part of the big picture
- 25. 25 Everyone has a part of the big picture
- 26. 26 Everyone has a part of the big picture
- 27. 27 Focus On All Parts of the Big Picture
- 28. Metrics Can Drive Behavior!! • Factory workers were measured on productivity as lighting was increased. • Researchers found that productivity increased when lighting was decreased also. • People will change their behavior when they think you are watching 28
- 29. Metrics Can Drive Behavior!! Test Team Productivity Example: • Testers required to write 15 cases per day and execute 25 per day • Blocking defect found in the first test. Testers kept running subsequent cases to meet their quota. 29
- 30. Metrics Can Be Misleading 30
- 31. Metrics Can Be Misleading 31
- 33. Oracles are Fallible Sometimes when things go right, we think we know why. But do we really? That the result went the way you wanted does not mean you can repeat it over and over. Simon Sinek “Start with Why” 33
- 34. Pretty Metrics vs. Meaningful Metrics 34
- 36. Let’s Just Automate It! • Automation is the ANSWER!! – What was the QUESTION? • Do you know what you’re trying to report? • Do your stakeholders know? • CONFUSION + NO CLUE = METRICS THAT ARE TRASH If you automate it, then all you get is….FAST TRASH! 36
- 38. Exercise 6: Measure This…. • What did you think I asked you to measure? • How did you start? • What did you use to measure? • Did you measure all the dimensions or one specific area? • What were your start/end points for the measurement? • Are all the tables the same size? Is your organization asking you to measure and report on things this way? If so, how do you get clear definitions? 38
- 39. 39
- 40. Use Consumable Metrics Scott Adams, http://www.dilbert.com 40
- 41. Where Do We Start? • Excellent testing starts by first questioning the mission. • Who is the client? • Who will use it directly/indirectly? • Who are the stakeholders? • What is our mission for this test? • What do people say about it? • What is its history? • Ask stakeholders – “May I ask a question?”. If no, then outline risks associated with proceeding with insufficient information. • What if the client doesn’t know? We might start by looking at a map of the system and asking questions about it. 41Michael Bolton, Better Software Magazine, Oct/Nov ‘08
- 42. Be Careful What you Ask… If I had asked people what they wanted, they would have said ‘a faster horse’. Henry Ford 42
- 43. Exercise 7: Let’s Define Metrics for a Sample Project 43
- 44. Retailer You work for a major retailer that sees the majority of their sales during the holiday timeframe in the U.S. This retailer has multiple channels for sales: in store, website, and mobile. It’s nearing the end of the year, and your team is called upon to test the releases that will support the holiday sales season, which starts the day after Thanksgiving. The team is also asked to verify that existing systems (with or without changes) will function accurately as well. 44
- 45. About Me 45 QA Director with over 20 years in IT: • Development • Project Management Office • Development Manager • Testing • Functional Testing • Test Environments • Software Configuration Management • Test Data Management • Performance Testing • Test Automation • Service Virtualization • Building Testing Organizations • Defining Processes, Methodologies, Measurement Mike Lyles
- 46. Correlation • Confusing concurrence with correlation – Consumption of Cheese is related to Deaths by Tangled Bedsheets • Confusing correlation with causation – Consumption of Cheese causes Deaths by Tangled Bedsheets Source: http://www.tylervigen.com/ 46
- 47. Does performance Matter? Source: gomez.com Implications for: Website owners; End users; Browser developers
- 48. 401% 319% 279% 220% 217% 196% 193% 193% 189% 185% 171% Market trends and Statistics Fastest Growing IOS & Android Markets by Active Devices Page load time (in Secs) Adoption trends of mobile devices across the world Average Page load time trends are evolving with the regards to users mobile adoption trends a 0 1 2 3 4 5 6 7 Argentina Iran India Brazil Turkey Mexico Chile Russia China United States Vietnam 2012 2013 Source: Google Analytics
- 49. 49 Industry wise performance Benchmarks Retail Industry Finance Industry Education Industry Healthcare Industry Media Industry Travel Industry 5.1 Secs 6.9 Secs 3.2 Secs 1.7 Secs 10.4 Secs 5.7 Secs Irrespective of the Industry type, Page load time of applications is always a critical area where the web developers should focus Below table shows the industry wise performance benchmark for Page loading time specifically for US based applications: Source: Compuware
- 50. Exercise 8: Why Do You Want to Measure?
- 51. Measurement vs. Metrics • A measurement is an observation that relates one thing to another, based on some model or theory, for the purpose of making some distinction. • The metric is the operation and the mathematical function that establishes (or “maps”) the relationship. Definition accredited to Michael Bolton Measurement is the art & science of making reliable observations. —Jerry Weinberg, Quality Software Management Vol. 2 51
- 52. Stop Counting! • Interviewer asks about hardest role • Actor says Hamlet 8,262 words • Othello is hard, has 941 fewer words • “How many words in King Lear?” • Actor cautions, “Well, now, I don’t want to give you the impression that it’s simply the # of words. Getting them in the right order is just as important.” Michael Bolton article “What Counts” http://www.developsense.com/articles/2007-11-WhatCounts.pdf 52
- 53. What is the Question? It’s so important for testers to change the question “Does this test pass or fail?” to a question that better addresses a possible threat to someone’s values: “Is there a problem here?” Taken from a Michael Bolton article “Issues About Metrics About Bugs” http://www.developsense.com/articles/2009-05-IssuesAboutMetricsAboutBugs.pdf Referring to a James Bach quote in article “Is There A Problem Here?” http://www.michaeldkelly.com/pdfs/IsThereAProblemHere.zip 53
- 54. 500 Defects!!
- 55. 55 Exercise 9 – Part 1 What do you define as “Good Metrics”
- 56. 56 Exercise 9 – Part 2 What do you define as “Bad Metrics”
- 57. 57 Exercise 9 – Part 3 What do you define as “Misleading or Misunderstood Metrics”
- 59. Analysis
- 60. Design
- 61. Development
- 62. Testing
- 64. Production
- 65. What Are the Experts Saying?
- 66. 66 Question 1: What do you feel are the major issues with measurement / metrics today in organizations?
- 67. 67 Q1: Major issues with measurement / metrics today in organizations? Many organizations seem to be trying to measure without considering the relationship between their measurement scheme and what they think they’re measuring. - Michael Bolton Bug counts, ratios, tests completed have no real connection with release readiness. Using these to make judgments or control projects is almost always dysfunctional – Doug Hoffman Teams think they recognize that they have multiple audiences, but fail to make appropriate adjustments when translating is needed. Thinking about how best seller books don’t end up best sellers in other markets is an analogy. - Christine Eliseev
- 68. 68 Q1: Major issues with measurement / metrics today in organizations? There are the cynical frightened testers who say yes to providing metrics that they know mislead their management. This is a phenomenon called pathetic compliance: when you do as you are told, not as you believe is reasonable and responsible. - James Bach One metric that, in almost any context, is particularly useless and misleading is test case counts. Stop counting test cases. Numbers are not magic charms. Stop abusing the dignity of mathematics! – James Bach
- 69. Q1: Major issues with measurement / metrics today in organizations? Humans will game the metrics, particularly when there are financial repercussions (e.g. tied to bonus or raise). – Paul Grizzaffi Judging developers on the number of bugs they 'produce’. – Lisa Crispin Testing organizations appear to create pretty charts and graphs because they think their clients want to see the information. They have told me that they don’t understand what the reports are trying to say BUT they ALL assume that someone else is looking at the metrics. – Paul Holland 69
- 70. Q1: Major issues with measurement / metrics today in organizations? The problem with a quick assessment is how much you can miss and then go on to make decisions and hold opinions with misinformation. Example - suppose you claim last month your team only had one critical defect found in production? One defect doesn't sound too bad but imagine the one defect is that the mobile app crashes for all users or no credit card process is working on your website. A quick glance of a metric or dashboard and you'd be off and planning with the wrong information. Karen Johnson 70
- 71. Question 2: What advice would you give to someone who works for an organization which uses the traditional numbers with bad metrics, and they want to invoke change? 71
- 72. Q2: Advice to someone that wants to invoke change? Try this exercise: • Take a number that has changed over some period of time. • Challenge each person around the table to come up with at least three dramatically different explanations for the change. • As a suggestion, remember that incompetence or outright malice in the measurement could provide an explanation. Michael Bolton 72
- 73. Q2: Advice to someone that wants to invoke change? When I encounter an organization that wants to change, I ask what story the stakeholders want to hear and work out how to describe that story. Measures/metrics may or may not be part of the story. – Doug Hoffman Bad metrics are helpful for those organizations who earn their bread and butter by selling test cases and because it makes too easy for them to manipulate or exaggerate the reality in a way that will give them more business and billing. - Lalit Bhamare 73
- 74. Q2: Advice to someone that wants to invoke change? Refuse to condone or contribute to stupid numbers. Always take the concerns and desires of your management seriously, but that doesn't mean they can order you to tell lies to them. If the manager says "I don't need the numbers for myself. It's MY boss who needs them." Then you can say "Look, I'm not going to help you mislead them. That is not one of the services I provide to this company." If you hear someone say "The numbers speak for themselves!" ask them whether teapots and chairs also talk to them. No number has ever "spoken for itself." Every claim about the meaning of a number is a story. James Bach 74
- 75. Q2: Advice to someone that wants to invoke change? Continue to report metrics but add a good story to the report. List the show stopping defects, new defects, risks, and issues impeding testing. Show mind maps to indicate coverage and show areas with bugs and issues. – Paul Holland Identify ways where metrics tell misleading or conflicting stories - things like there being a relatively high number of tests that don't actually result in a lot of test coverage or places where metrics are being compared to draw incorrect conclusions. – Andy Tinkham I work with this issue often. First you have to recognize how much influence you have in an organization - it is unlikely to overthrow a practice such as the following metrics overnight. Sometimes, I try to work in non-metric forms of measurements – Karen Johnson 75
- 76. Question 3: What do you consider is the greatest impact to organizations that are stuck in the “bad metrics cyclone”? 76
- 77. Q3: Greatest Impact? People often say that imperfect measurement is better than no measurement, which to me is a little like saying that scoring an own goal is better than not scoring a goal at all. Let us remember that numbers, like pictures, are illustrations of the story; they are not the story. So instead of simply accepting numbers as an accurate picture of things, let’s ask “What’s wrong with this picture? What might be wrong with it?” Michael Bolton 77
- 78. Q3: Greatest Impact? It promotes a culture of cynicism and deception. A culture without true self-respect. - James Bach Making poor decisions because the wrong information or the wrong view of information was used. – Karen Johnson Metrics should never be used to judge performance of individuals or teams. They are only tools to help teams know whether what they are trying now is helping or not. – Lisa Crispin 78
- 79. 79 The GQM Model
- 81. Practicing the GQM 1. Determine the goals of the stakeholders and/or project team. 2. Define, from each goal, which question must be answered to determine if the goals are being met. 3. Document what must be measured to accurately answer the question. 81
- 82. GQM - Example • Goals – Reduce total cost of development – Reduce total cost of testing-effort – Reduce the number of new feature bugs • Questions – How frequently are releases moved to production? – Which functional areas have the most defects? – How long does it take to repair defects? – Which areas have the highest re-work rates? 82
- 83. Exercise 11: Applying the GQM Answer the following for the upcoming project: 1. What you need to measure? 2. Who are your stakeholders? 3. Using the GQM – what are your: • Goals • Questions • Measures / Metrics 4. How will you present your measurements? 83
- 84. Banking – Year End You have been hired to lead the testing efforts for a very large bank. Your first project will be to test the new application developed to support the end of year reporting for financials and the entire bank reporting related to year-end. This information will be used to report banking financials to the stock market, providing year end statements to banking customers, as well as reporting bank yearly financials to the government for tax purposes. 84
- 96. Kanban Approach
- 97. Whiteboard • Used for planning and tracking of test execution • Suitable for use in waterfall or agile (as long as you have control over your own team’s process) • Use colors to track: – Features, or – Main Areas, or – Test styles (performance, robustness, system) Credited to Paul Holland 97
- 98. Whiteboard • Divide the board into four areas: – Work to be done – Work in Progress – Cancelled or Work not being done – Completed work • Red sticky notes indicate issues (not just bugs) • Create a sticky note for each half day of work (or mark # of half days expected on the sticky note) • Prioritize sticky notes daily (or at least twice/wk) • Finish “on-time” with low priority work incomplete 98 Credited to Paul Holland
- 99. Sticky Notes • All of these items are optional – add your own elements • Use what makes sense to your situation – Charter Title (or Test Case Title) – Estimated Effort – Feature area – Tester name – Date complete – Effort (# of sessions or half days of work) • Initially, estimated -> replace with actual 99 Credited to Paul Holland
- 100. Actual Sample Sticky Charter Title Tester Area Effort 100 Credited to Paul Holland
- 101. Whiteboard Example 101Credited to Paul Holland
- 102. Whiteboard Example 102Credited to Paul Holland
- 103. Metrics Are NOT Always About The Product • Periodically ask your stakeholders questions • Keep the same questions and monitor change • Give them a forum to provide constructive feedback 103
- 104. …and they lived happily ever after. Questions?
- 105. Mike Lyles QA Director, Bridgetree Twitter: @mikelyles mikewlyles@gmail.com www.MikeWLyles.com www.TheDriveThruBook.com
- 106. Appendix
- 107. Example 1 - Retailer You work for a major retailer that sees the majority of their sales during the holiday timeframe in the U.S. This retailer has multiple channels for sales: in store, website, and mobile. It’s nearing the end of the year, and your team is called upon to test the releases that will support the holiday sales season, which starts the day after Thanksgiving. The team is also asked to verify that existing systems (with or without changes) will function accurately as well. 107
- 108. Example 2 - Mobile App You have been hired as a consultant by a software development company. They have implemented a new app that can be used by educational organizations (schools, universities, executive training programs, etc). There are at LEAST two other related apps being developed by other companies. And it is very likely that the first company to produce the app will get the majority of downloads. You have been asked to test the application within a 3-week timeframe so that they can get their product into production. 108
- 109. Example 3 – Medical Device Knowing your testing background, a major medical device provider has contacted you to test a device they have created to diagnose, monitor, regulate, and remediate high blood pressure. There have been devices in the medical field that provide one of the various services this device will deliver. However, there is not one currently that will do all of the same functions with one device. This device could change the way high blood pressure is remediated. However, there is a high risk if the remediation function of the device causing issues with patients. 109
- 110. Example 4 – Banking – Year End You have been hired to lead the testing efforts for a very large bank. Your first project will be to test the new application developed to support the end of year reporting for financials and the entire bank reporting related to year-end. This information will be used to report banking financials to the stock market, providing year end statements to banking customers, as well as reporting bank yearly financials to the government for tax purposes. 110
Editor's Notes
- #4: All good stories start with this, right?
- #7: Exercise 1: Everyone take out a piece of paper – write down your story. If we just met, and you wanted us to know a LOT about you in 5 mins or less, what would you say about yourself? What makes you UNIQUE? What should we remember about you if we meet you again in a year on the street?
- #9: Let’s count the number of ______ in this room……. Give everyone something to measure
- #10: Measuring just because you can doesn’t always give results that you need to use. The information you gathered may be useful for somebody else but it isn’t of any value to you, me, or anyone else in this room When you are building metrics, you have to know if what you are gathering and counting has ANY value to ANY ONE at ALL.
- #11: 1 Conference 4 Days 2 Days for Tutorials 20 Tutorials 1 Speaker Meet & Greet 2 Conference Days 4 keynotes 37 sessions 1 Welcome Reception / Expo 48 Speakers 200+ Attendees
- #12: Guess our age, my height, my weight, favorite color, favorite food, year I graduated high school, college, number of kids that I have (if I have any),
- #13: 8/29/71 – I was born 2 kids – 1 boy 13 1 girl 8 Graduated high school 89, College 93 First job was making $20,000 Started working for Lowe’s in 1994 as a programmer – worked there for 19.5 years I’m 6’1 Green Eyes and Brown hair And…..well some metrics are best left un-shared
- #14: 4-8-15-16-23-42 (number that kept coming up in LOST TV show) 365 – days in the year 24 – hours in a day 7 – days in a week **** NOTE:: would 24 or 7 stood out if I had not used 365 first? 365 only has a few associations, but 24 could mean the Keiffer Southerland TV show, and 7 could mean 7-up, 7 dwarfs, 7 wonders of the world…..it’s all relative There are 292 Bones in the body There are 7 continents There are 196 countries There are 8 planets There are 50 states When numbers can tell ANY story that is when we have room for error. Take “1” for example. It makes it hard to define what your’e talking about if you put “1” there. Is it 1 brain? 1 house? 1 earth? 1 car? 1 is the loneliest number…. 26 January : Australia Day 7 March: Labour Day 01 November: Melbourne Cup Day
- #15: If this is true – how can you make your organization, your stakeholders, your team, your customer LOVE YOU by simply telling them your story, the story of the work that is going on, and the story of the quality of the product you are supporting?
- #16: Example: Bob says “I’m a tester – I work for company X, I am a man, a father, a husband, and I like bananas”. All these things make up BOB but there may be 10 other people in the class that have the same details.
- #20: If you’re like me, you step on the scales multiple times per week You are looking at yourself closely – daily – sometimes hourly You are measuring yourself based on what you do every day – how much you eat, how many calories you take in, whether you eat bad things or good Your metrics may be the scale, tight or lose clothes, feeling sluggish, feeling energetic, looking at yourself in the mirror You measure how much you exercise and you measure when you go to sleep and when you wake up every day Our smartphones and mobile devices are all creating devices, apps, and software to help us better measure our daily reports
- #21: Physical health vs testing status. Overall health not day specific. Consistent performance graded at the end of year Not daily measurement of test cases execution. What I do today will impact me tomorrow. But thats not the big picture. Big picture is - end of year doctor's runs his tests and his results wont tell me if i ate right this day or not but more about overall health of the system after the year long of taking care of the body
- #22: Physical health vs testing status. Overall health not day specific. Consistent performance graded at the end of year Not daily measurement of test cases execution. What I do today will impact me tomorrow. But thats not the big picture. Big picture is - end of year doctor's runs his tests and his results wont tell me if i ate right this day or not but more about overall health of the system after the year long of taking care of the body
- #24: Puzzle
- #25: Puzzle
- #26: Puzzle
- #27: Puzzle
- #28: Puzzle
- #29: The hawthorne effect taught us a valuable lesson – people behave differently when they know they are being monitored Make sure that your metrics are not causing teams to compete or to try to meet a goal. Meeting goals is good – but make sure the positive outcomes are valid.
- #30: I had this situation happen to me in real life. We set a metric for tracking number of cases written/executed, and the team made sure they hit the mark. However, when there was a blocking defect that the tester knew would hold up 20 more cases, they continued to run and log defects for the other tests so that they met their quota for the day.
- #31: Just because you have a metric doesn’t mean its accurate. Make sure that what you are reporting is not misleading or misrepresenting the true status of the deliverables. And make sure you’re not telling one story and your stakeholders are getting another.
- #32: I took this picture in a public bathroom Don’t worry, I washed my hands afterwards But the issue here is that we don’t know what 0.35 is…..is that GOOD? Is it BAD? Is it 35% clean? Or 35% dirty? Or even 35%? I would have a hard time knowing how to “do my part to improve this situation” when I don’t know what the scale is and what GOOD is.
- #33: Metrics can be extremely misunderstood. If you don’t read them correctly, sometimes you miss something that is wrong with them – and it’s right in front of your face.
- #36: Know your sample data – obviously I was NOT called to partake in this data gathering exercise. And frankly it makes me SAD But seriously, probably no one in this room was asked if they are happy or not, but everyone in the USA seems happy from this show of data. I know of only three people hat has truly given this metrics – Pherell Williams – he had the song “Happy”. Bobby McFerron said “Don’t worry, be happy” and the Patridge Family sand “Come on Get Happy”. It’s important when you’re presenting metrics and showing output that you are clear on how you gathered it and the size of the data you pulled.
- #37: Automation is the answer only when you understand the question. If you don’t know what you’re trying to report and you or your stakeholders can’t define it, you likely have TRASH Therefore the automation of that doesn’t solve your issue – you just end up with FAST TRASH
- #40: McDonalds just wants to make money Consultants that make momey on # cases written, # cases executed, % pass or fail, or % complete MAY not care if its helping your company
- #41: It is important to not only know who you are stakeholders are for your reporting, but to also make sure that the information you are providing can be understood. You can lose an audience very quickly if you don’t focus on the customer who will be reading the report.
- #42: Testers should be informing the teams of risk Use your testing experience to guide and direct the stakeholders Metrics that are invalid contain risks
- #44: It is important to not only know who you are stakeholders are for your reporting, but to also make sure that the information you are providing can be understood. You can lose an audience very quickly if you don’t focus on the customer who will be reading the report.
- #46: No, I didn’t accidentally go back to my “About me page” And no, I’m not a narcissist that just wants to talk about ME ME ME all the time I brought this back to see if you noticed something in my biography that was subtle I’m wondering how many of you assumed that the name tag “Hello My Name” was a regular nametag that we have seen many times. Did you notice that instead of saying “My name IS Mike Lyles” it says “My name ISN’T Mike Lyles” ----- Meeting Notes (3/20/16 19:35) ----- 1:00
- #47: Just because things are related, and may even seem to change at the same rate, it does not necessarily mean they are related or that they should be reported on together.
- #48: Another ‘health check’ that is similar to the yearly checkup with a doctor is performance monitoring. It is important to understand if your systems are responding in the speed at which customers will remain engaged and coming back. Studies show that customers get very frustrated with slow systems. Many will decide immediately not to come back. Many will look at your competitors for support. These are metrics that cannot be pulled from test creation and execution, but they surely must be taken into consideration when you’re reporting the results.
- #49: Knowing your metrics and stakeholders is critical, however, it’s also important to know the Market Trends. In this study, you can see the countries that are moving into the mobile markets very aggressively. With this rapid move to mobile comes a need for faster response times and page load times. This should be taken into account when you are measuring the success of your project.
- #50: Along with Market Trends comes the need to understand your industry. If you’re in retail, it has been found that the allowable wait time for page loads is 5.1 seconds. People will wait longer for media to load, as we have grown accustom to waiting for movies and music to load. But with critical systems, such as healthcare, the response times need to be faster. Knowing your company’s domain and the research that is being done across all the domains is a way that you can improve not only your measurement program but your IT deliverables in general.
- #52: This is from the internet, widely used source, don’t think from bolton. Seen many times in wikipedia, etc…
- #53: Too many times we are counting ---- “how many automated cases vs. ‘manual’”, “how many test cases written today”, “how many executed today”, “how many defects have we found today”, “number of steps in a test case”. This is data that may be useful to the testing team but is hardly ever a critical reporting fact for the stakeholders or the project team. (remember the health discussion – weighing on a scale vs. the overall health in the yearly checkup with the doctor). Make sure you don’t ‘train’ your customers and stakeholders to view the low level “behind the scenes” metrics as ones that are critical to them. Keep these for yourself – give them what they need in the context that they need it. FULL STORY BELOW: In a relatively obscure Monty Python sketch, an interviewer asks a seasoned but clearly incompetent actor about the hardest role in the theatre. The veteran responds that the answer must be Hamlet—Hamlet has 8,262 words. Othello is hard, too—but it has 941 fewer words than Hamlet. The interviewer learns quickly: “How many words did you have to say as King Lear?” But the actor cautions him, “Well, now, I don’t want to give you the impression that it’s simply the number of words. Getting them in the right order is just as important.”
- #54: Too many times we are counting ---- “how many automated cases vs. ‘manual’”, “how many test cases written today”, “how many executed today”, “how many defects have we found today”, “number of steps in a test case”. This is data that may be useful to the testing team but is hardly ever a critical reporting fact for the stakeholders or the project team. (remember the health discussion – weighing on a scale vs. the overall health in the yearly checkup with the doctor). Make sure you don’t ‘train’ your customers and stakeholders to view the low level “behind the scenes” metrics as ones that are critical to them. Keep these for yourself – give them what they need in the context that they need it.