Bad metrics and how to spot them - Music City Agile 2018
- 1. Bad agile metrics and how to find them
- 2. Agenda âą Introduction âą Level set on what metrics are âą Why we use metrics âą How bad metrics can impact your team âą Bad agile metrics âą What good metrics look like âą Takeaways âą Q&A
- 3. Introduction Faye Thompson Agile Coach/Scrum Master Cardinal Solutions Charles Husemann Agile Coach/Scrum Master Cardinal Solutions
- 4. #MusicCityTech #MusicCityAgile Also weâve updated our privacy policyâŠ.
- 5. What is a metric?
- 6. Metric: A standard of measurement
- 7. Agile Metric: Empirical measurement of a dimension of a teams performance
- 8. Who are metrics for? âą The development team âą Management âą Project stakeholders âą Anyone walking by the team space
- 9. Why are metrics important?
- 10. Why metrics are important âą Give you the ability to Measure - Inspect - Adapt âą Tell the âstoryâ of your team âą Promote transparency and visibility âą Tracks sprint/release progress âą Creates accountability âą Helps to achieve and optimize predictability
- 11. The power of metrics âą Sharpen team focus âą Provide measurable goals âą Promote understanding of goals âą Motivational tool âą See trends in work (good and bad) âą Drives team behavior (good and bad)
- 12. Why sniffing out bad metrics is important
- 13. The impact of the misused metrics âą Disruptive to team âą Counterproductive to intent âą Set unmanageable/unrealistic expectations âą Discourages cooperation âą Team worries about visibility of metrics âą Encourages the wrong behavior
- 14. Two psychological effects to keep in mind when talking about metrics âą Hawthorne Effect âą Streetlight Effect
- 15. Hawthorne Effect The alteration of behavior by the subjects of a study due to their awareness of being observed.
- 17. How else might this show up in your team?
- 18. Streetlight Effect A type of observational bias that occurs when people are searching for something and look only where it is easiest
- 20. How would you fix this on your team?
- 21. Bad agile metric smells With great power comes great responsibility
- 23. Leaderboards
- 24. Competition within the team âą Impacts work selection and estimation âą Complexity of work âą Smaller stories âą Sandbag estimates and work âą Discourages cooperation by focusing on the individual rather than the team
- 25. Team vs Team
- 26. Team comparison
- 27. Team size and composition could be different
- 28. Sprint length
- 29. Point sizes differ by team
- 31. Metrics out of context
- 32. Metrics without context 2016 Dark Side Challenge 2017 Dark Side Challenge
- 33. 0% 20% 40% 60% 80% 100% 120% 140% Sprint 31 Sprint 32 Sprint 33 Sprint 34 Sprint 35 Sprint 36 Sprint 37 Sprint 38 Sprint 39 Sprint 40 Sprint 41 Sprint 42 Sprint 43 Sprint 44 Sprint 45 Sprint 46 Sprint 47 Sprint 48 2017 Sprint Completion Performance Actual Performance Target Performance
- 34. Metrics live within a context âą Internal blockers âą Team member(s) out sick âą Holidays âą Influx of bugs âą Unplanned work âą External blockers âą BU availability âą Technical/Infrastructure issues âą Weather
- 35. Incentivizes the wrong thing
- 37. Incentivizes the wrong thing âą Avoid creating âHeroesâ âą Goal is a sustainable and predictable pace â not to see who can get the most points done in a sprint âą Is this measuring a desired behavior?
- 38. Other bad signs bad metrics âą Measure bad smells/non-agile things âą Lack sufficient data to see a trend âą Attempts to measure abstract concept
- 39. Good metrics
- 40. What do good metrics look like? âą Clear intent âą Not gameable âą Show progress over time âą Identify potential issues and areas for improvement âą Are understood by the entire team, not just the Scrum Master âą Smallest metrics that can possibly work âą A good metric for one team may not be applicable to another team
- 41. Examples Are these good or bad?
- 42. Sprint Burndown Chart 65 65 62 62 62 58 58 58 58 16 0 0 3 3 3 11 11 11 11 0 10 20 30 40 50 60 70 80 StoryPoints Transformers Sprint 3 Burndown Remaining Total Story Points Completed
- 43. Velocity Chart 107 246 126 173 193 318 250 291 176 227 151 133 104 139 155 100 265 355 116 0 50 100 150 200 250 300 350 400 Team Awesome Velocity (Story and Defect Points)
- 44. Points per developer per sprint
- 45. Backlog Health 0 2 4 6 8 10 12 Sprint 31 Sprint 32 Sprint 33 Sprint 34 Sprint 35 Sprint 36 Sprint 37 Sprint 38 Sprint 39 Sprint 40 Sprint 41 Sprint 42 Sprint 43 Sprint 44 Sprint 45 Sprint 46 Sprint 47 Sprint 48 Product Backlog Health
- 46. Defects found per tester 0 2 4 6 8 10 12 14 16 18 20 Sprint 10 Sprint 11 Sprint 12 Sprint 13 Sprint 14 Sprint 15 Wally Sue Diana
- 47. WIP vs. Velocity 0 20 40 60 80 100 120 140 Sprint 12 Sprint 13 Sprint 14 Sprint 15 Sprint 16 Sprint 17 Sprint 18 Sprint 19 Sprint 20 Sprint 21 Work in Process Points per Week
- 48. Takeaways: âą Bad Metrics: âą Pit team members against each other âą Pit teams against each other âą Be obscure âą Measure bad practices âą Encourage bad behavior âą Good Metrics: âą Are clear and concise âą Encourage team behavior and cooperation âą Help the team identify areas for improvement and track progress towards goals âą Tell part of the story of your team âą Act as a conversation starter
- 49. Open Discussion
- 50. Faye Thompson fthompson@cardinalsolutions.com @agilefaye Charles Husemann chusemann@cardinalsolutions.com @chusemann Cardinal Solutions Group Agile Advisory Services - CMH Thank You
- 51. Contact information Faye Thompson - fthompson@cardinalsolutions.com @agilefaye Chuck Husemann â chusemann@cardinalsolutions.com @chusemann
Editor's Notes
- #2: FAYE START Bad agile metrics now â Resistors an hour and a half ago Everything else â not here
- #4: Welcome to the Bad agile metrics and where to find them
- #5: Hashtag slide
- #7: So what is a metric? â Hereâs the basic definition straight from Merriam-Webster
- #8: I use empirical here for two reasons âFirst because it states that things are verifiable by observation/experience and second because itâs a word that the Scrum.org people like to use.
- #9: FAYE STOP
- #10: CHUCK START Just so I know â how many people are currently working on agile teams? If I use a term you arenât familiar with or donât know please raise your hand and let me know.
- #11: Metrics are the foundation of the empirical process Itâs also important as metrics support the scrum values - Focus, Openness, commitment, courage, Respect,
- #15: Whenever you deal with metrics there are two psychological effects to worry about. CHUCK STOP
- #16: FAYE START Think about it this way - if you know you are going to have your blood sugar monitored you are going to stay away from sweets Or if you know thereâs a copy ahead â you slow down
- #17: Sandbagging estimates is the most obvious one but it could also show up by your team not tracking/logging defects
- #21: FAYE END Root cause analysis â is the metric supposed to cover this? Do we need another metric to suss something out?
- #22: CHUCK START
- #24: https://www.youtube.com/watch?v=oyVksFviJVE Gilfoyle vs Dinesh
- #25: CHUCK END
- #26: FAYE START
- #27: Bad to compare direct numbers but could compare trends
- #28: Team Size
- #29: Schedule
- #31: Definition of done The team starts to frantically work harder, producing lower quality software or burning out The team changes the definition of Done to allow for faster velocity, but lower quality product or product that is actually not finished and ready to ship The team changes its reference stories, doing the same amount of work but clocking a higher velocity The team loses its passion for driving higher velocity and doing better, leading to worse performance, as its intrinsic motivation to improve is replaced by extrinsic motivation and stress from management, but in general, for a lot of work, especially âthinkingâ work like software development, intrinsic motivation, which is a passion that comes from within the worker, is generally a better motivator and driver of performance than extrinsic motivation, such as punishment or reward from management. Furthermore, extrinsic motivation reduces intrinsic motivation) FAYE STOP
- #32: CHUCK START
- #34: Team made the switch to 3 week sprints at the start of this year which so we are learning how much work we can actually do in a sprint. We will have more change the next sprint as we are losing a developer
- #39: CHUCK END
- #40: FAYE START
- #41: FAYEÂ STOP
- #42: CHUCK START So now that weâve given you an overview letâs look at some real world examples and see if an tell which ones are good and which ones are bad.
- #43: FAYE This is a good team as thereâs more than meets the eye
- #44: CHUCK
- #45: FAYE
- #46: CHUCK
- #47: FAYE
- #48: CHUCK
- #49: FAYE