Improving the development process with metrics driven insights presentation
- 1. Improving the Development Process with Metrics-Driven Insights oscon.indeed.tech | @IndeedEng | github.com/indeedeng
- 2. Hi, I’m Jack. Jack Humphrey, VP Engineering go.indeed.com/jack | @youknowjack
- 4. 60 countries 30 languages 200M unique visitors 20M jobs
- 5. Offices around the world 4 5 New York Stamford Toronto São Paulo Austin Seattle San Mateo San Francisco Dublin London Amsterdam Dusseldorf Paris Hyderabad Bangalore Singapore Sydney Tokyo Zurich Brussels Scottsdale
- 7. Indeed Engineering: move fast and try things
- 8. Don’t bet on a small number of “great ideas”
- 9. Bet on exploring lots of ideas as quickly as possible
- 10. Exploring lots of ideas as quickly as possible 1 Hire great people 2 Give them ownership and autonomy 3 Develop great tools
- 11. We’ve open sourced two of these tools Proctor A/B Testing Framework Imhotep Data Analytics Platform http://oscon.indeed.tech
- 12. - Enables rapid exploration & analysis of large time-series datasets - Query language (IQL), web UI, and distributed backend Imhotep: data analytics platform fastefficientscalable github.com/indeedeng/imhotep
- 13. Up to 5 million queries/week at Indeed across ~1500 datasets Most popular dataset has 39B events over the last year - Last 30 days: 25K distinct queries invoked 2.2M times by over 1200 users - ~95% of invocations are by a few automated users Imhotep: scalable, efficient, fast
- 14. Most popular dataset: 39B events, 384 fields - 5.7TB on disk (146 bytes/event) No need to sample — use all the data Imhotep: scalable, efficient, fast
- 15. Most popular dataset: queried 2.2M times in last 30 days - Median response time 159 milliseconds - 95% of queries < 2 seconds - Median uncached response time 5.7 seconds - Median response time, uncached 365-day queries: 37s Imhotep: scalable, efficient, fast
- 16. These tools enable our approach to development Deliver MeasureLearn
- 17. We can use this approach to improve our processes 01 Measure everything 02 Ask lots of questions in order to learn 03 Based on those learnings, make changes (deliver) 04 Then measure to confirm improvement
- 18. 01 Measure everything 02 Ask lots of questions in order to learn 03 Based on those learnings, make changes (deliver) 04 Then measure to confirm improvement We can use this approach to help people improve
- 19. Is measuring process and people a good idea?
- 20. * proceed with caution Yes.*
- 21. Goodhart’s Law When a measure becomes a target, it ceases to be a good measure.
- 22. “I am not a number.”
- 23. Measures aren’t inherently bad… it’s how you use them.
- 24. The metrics should serve the team. The team shouldn’t serve the metrics.
- 25. Process improvement: an example Measure LearnDeliver
- 26. - Everything that happens in our products - Everything that happens in our process - Git commits - JIRA issue updates - production deploys - wiki edits - and more... Measure (a.k.a. “put it in Imhotep”)
- 27. - We translate each string change or new string to 30 languages. - We use a custom JIRA issue type to track: 1 per language. - Verification has been historically tedious... - So let’s measure how long we spend verifying translations. Process improvement example: translation verification
- 28. from jiraactions 2017-01-08 2017-04-02 where issuetype = 'Translation' AND prevstatus = 'Pending Verification' AND status != 'Pending Verification' AND project = 'LOREM' group by time(1d) select timeinstate/86400 /* days pending */ How long are translations in “Pending Verification”?
- 29. from jiraactions 2017-01-08 2017-04-02 where issuetype = 'Translation' AND prevstatus = 'Pending Verification' AND status != 'Pending Verification' AND project = 'LOREM' group by time(1d) select timeinstate/86400 /* days pending */ How long are translations in “Pending Verification”?
- 30. from jiraactions 2017-01-08 2017-04-02 where issuetype = 'Translation' AND prevstatus = 'Pending Verification' AND status != 'Pending Verification' AND project = 'LOREM' group by time(1d) select timeinstate/86400 /* days pending */ How long are translations in “Pending Verification”?
- 31. from jiraactions 2017-01-08 2017-04-02 where issuetype = 'Translation' AND prevstatus = 'Pending Verification' AND status != 'Pending Verification' AND project = 'LOREM' group by time(1d) select timeinstate/86400 /* days pending */ How long are translations in “Pending Verification”?
- 32. from jiraactions 2017-01-08 2017-04-02 where issuetype = 'Translation' AND prevstatus = 'Pending Verification' AND status != 'Pending Verification' AND project = 'LOREM' group by time(1d) select timeinstate/86400 /* days pending */ How long are translations in “Pending Verification”?
- 33. Verification time is adding up
- 34. - Can we dig into the data to better understand it? - What other information is needed to interpret? - What are the sources of noise? - Do we need to iterate on the measurement itself before it is generally useful? Be skeptical and question all measurements.
- 35. from jiraactions 2017-01-07 2017-04-29 where issuetype='Translation' AND prevstatus='Pending Verification' AND status != 'Pending Verification' AND project = 'LOREM' group by time(1d) select distinct(issuekey) /* number of issues */ What’s the issue volume over that timeframe?
- 36. What’s the issue volume over that timeframe?
- 37. What’s the cumulative issue volume over that timeframe? 18 329
- 38. - Good measurements + good questions = learning - We now have a measurement that supports our hypothesis, and we can prioritize improvement. Learn and prioritize
- 39. - Better way: deploy translations separate from code - Does new process reduce verification time? - Project “LOREM”: old process - Project “IPSUM”: new process Translation verification: there is a better way
- 40. from jiraactions 2016-09-15 2017-02-28 where issuetype = 'Translation' AND prevstatus = 'Pending Verification' AND status != 'Pending Verification' group by project in ('LOREM','IPSUM') select percentile(timeinstate, 90) Let’s look at 90th percentile time in “Pending Verification”
- 41. The new process does look faster! Worth implementing in LOREM, then we’ll measure again 12 days 1.8 days
- 42. Helping people improve with metrics
- 45. Guard against Goodhart’s law 1 We use hindsight as a starting point for discussion 2 We constantly remind ourselves: don’t treat as targets.
- 46. Resolved: 100, Reopened: 30 - “Productive, but attempting to ship a lot of buggy code” Be skeptical. Dig into data. - Only 10 actual bugs Conversation can produce new ideas for individual and team Example conversation: how’s my quality?
- 47. Measure, question, and learn. It can work for process and people.
- 48. Tracking the Issue Tracker
- 49. Hi, I’m Kevin. Kevin Binswanger, Software Engineer
- 50. Known unknowns: quality 1 Reopens 2 Escaped bugs 3 Time and Severity
- 51. Known unknowns: engagement 1 How many committers 2 How much they are committing 3 Bus Factor
- 52. Known unknowns: velocity 1 Committed > Reviewed > Verified 2 Inside the team vs Outside the team 3 Places we can automate
- 53. Imhotep Dataset
- 55. January 1, 2016 to May 1, 2017 3.9 million actions
- 56. January 1, 2016 to May 1, 2017 1.6 million actions
- 57. #4 will surprise you! We Imported All the Activity from the Apache JIRA Instance, You’ll Never Believe What Happened Next! http://go.indeed.com/apachejira
- 58. Share what you learn about Apache projects: Tweet @IndeedEng with #imhotep #apache http://go.indeed.com/apachejira
- 59. Questions?
- 60. Share what you learn about Apache projects: Tweet @IndeedEng with #imhotep #apache http://go.indeed.com/apachejira oscon.indeed.tech | @IndeedEng | github.com/indeedeng