Taking the Guesswork Out of Performance Budgets
2 likes651 views
The document discusses taking the guesswork out of performance budgets by tying technical performance metrics to business outcomes. It recommends a 3 step approach: 1) Identify a key "hero" business metric. 2) Tie a performance budget to incremental improvement in that business metric. 3) Focus on optimizing the slowest user experiences. Measuring both business and technical metrics together at the individual user session level allows understanding how changes in performance impact the business.
Taking the Guesswork Out of Performance Budgets
- 1. Taking the Guesswork out of Performance Budgets Buddy Brewer — @bbrewer
- 3. Closed Won
- 4. Business Bookings, Revenue Sales Rep Quota retirement, Cash in pocket
- 5. $124B
- 6. What’s the average number of calls my reps make each week? How many POCs started this month? What is the total bookings in outstanding proposals? Who sent the most emails this week? What percent of demos convert to POCs?
- 7. What’s the average number of third party calls in my pages? How big is the hero image on my home page? What’s the median time to first contentful paint on mobile? What are my top 10 slowest URLs? What percent of pages load in over 4 seconds?
- 8. It’s critical to know how a change in performance will change your business
- 9. Measure business and technical metrics together.
- 10. ✘ faster is better ✔ how much faster and how much better
- 11. 1. Identify a hero metric 2. Tie a performance budget to a business outcome 3. Focus on the slowest experiences Consider 3 Steps
- 12. Identify a hero metric 1
- 13. Domain Lookup, Connect Time, Connect End, SSL Handshake, Response Start, Response End, DOM Loading, DOM Complete, DOM Interactive, Start Render, First Contentful Paint, First Meaningful Paint, User Timing Metrics, DOM Content Loaded, Load Event, Time to Interactive
- 15. Tie technical and business metrics together at the visit level session id| … |load|fcp|fmp| … | session len | converted?
- 16. Identify a single desired business outcome that is captured by a metric
- 17. Look for technical metrics that significantly impact the business metric
- 20. Considerations for random forest classifiers Handles unbalanced data sets well Doesn’t require as much tuning as other methods Will tell you something about itself Watch out for metrics that aren’t collected broadly
- 21. Tie a Performance Budget to a Business Outcome2
- 23. Derive performance budget from incremental business improvement Suggestion: Start with 10% improvement on the business metric
- 25. Segment by key user populations Country Connection type Device type
- 26. Make a lot of small improvements
- 27. Focus on the slowest experiences 3
- 28. Fix the metric value and watch the percentile Fix the percentile and watch the metric value Optimize Monitor vs
- 31. Ranking by multiple factors Dim: entry_page_type Val: item pages Dim: entry_url Val: https://example.com/page1234 Dim: entry_page_type Val: store locator pages Dim: country Val: US Dim: device_type Val: iPhone
- 32. Re-baseline performance budget at least quarterly
- 33. It’s critical to know how a change in performance will change your business
- 34. Start when you get back to work. Ask your business partners what metrics they use to define business success. Include these in your RUM data.
- 35. Resources Project Jupyter: https://jupyter.org/ Sklearn’s Random Forest Classfier: https://scikit- learn.org/stable/modules/generated/ sklearn.ensemble.RandomForestClassifier.html Example code: https://gist.github.com/bbrewer/ 17b29c0bbfb77dc76f5bbaa54c2c7ea4