Progressive Feature Rollout
3 likes525 views
The document outlines strategies for the progressive rollout of new features, emphasizing the roles of product owners, software engineers, and users in the development process. It discusses the importance of A/B testing, feature toggling, and metrics tracking using tools like Google Analytics and backend systems to assess user engagement and feature effectiveness. Key takeaways include the need for robust statistical evaluation and proper planning during feature deployment to minimize risks and enhance user experience.
Progressive Feature Rollout
- 1. Progressive Feature Rollout PHP User Group Milano 2017.06.21 Fabio Mora mail@fabiomora.com http://fabiomora.com
- 2. 1 / Discovery: idea and validation
- 5. Different roles, different needs Product Owner ● Give users features ● Validate ideas ● Short feedback loop ● Design right prototypes Software Engineer ● Build cheap prototypes ● Put dev time in the right features ● Keep tech debt low User ● Have the site available ● Use a coherent service ● See interesting features
- 6. Heads to...
- 7. Trade-off? Or even make the cheapest thing... https://martinfowler.com/bliki/DesignStaminaHypothesis.html “If the functionality for your initial release is below the design payoff line, then it may be worth trading off design quality for speed; but...”
- 9. A look to the metrics using Google Analytics ● Google Analytics out of the box ○ Sessions, Page Views, Users, Bounce Rate, Session Duration… ■ Reports: Audience, Overview. Then “Select a metric” section ○ Custom Events ■ Reports: Behaviour, Events, Overview. Then browse by “Event Action” ● Click-Through Rate (CTR) definition might help us ○ Ratio of the whole user traffic who click on a specific Call To Action (CTA) ○ CTR = Clicks / Page Views * 100
- 10. Category: register Action: credential-start Category: register Action: facebook-landing Category: impression Action: master OR feature-fb
- 11. Using Google Analytics events
- 13. A quick prototype with feature branches master feature-a Branch git checkout master git checkout -b feature-a Merge in master git rebase master git checkout master git merge feature-a https://martinfowler.com/bliki/FeatureBranch.html
- 15. A really really simple way to A/B test A {0, 1} random toggle and a X-Feature HTTP Header
- 16. A really really simple way to A/B test A Template Engine condition and GA events
- 17. A really really simple way to A/B test 50% vs 50% ...test it!
- 18. Going further with A/B Testing Unit of Division Example Pro Cons Event Page View Simplicity Randomness UX Inconsistent Fingerprint PHPSESSID, Cookies UX consistent by channel Low randomness Condition User Login UX Consistent Low randomness Not always applicable
- 19. A/B Testing: key points ● Unit Of Division ○ Is the only thing we’re allowed to design ○ Indipendent, homogeneus, representative (of the whole traffic) ● Evaluation Metric ○ Depends on the feature (e.g. CTR) ○ Other: time on page, loading time, first byte (avg, median, quantiles) ○ Focus on a few metrics!
- 20. A/B Testing: key points ● Significant Improvement ○ Tangible improvement from business point of view ● Sample Size ○ Repeatability: statistical phenomenon, variability, robustness, multiple runs ○ Significance: regarding a probability model
- 21. Refer to Math ● A/B Testing ○ Bernoulli Distribution fits {0, 1} events ○ Parameters: confidence, power, sample size, tangible improvement ○ Here Statistical Confidence (interval) is close to Statistical Repeatability ○ http://www.evanmiller.org/ab-testing/sample-size.html ● A/B/C/D… Testing ○ Way more complex ○ https://en.wikipedia.org/wiki/Bonferroni_correction
- 22. A bit of help...
- 23. Aside note: randomness and PHP ● rand() ● mt_rand()
- 24. rand()
- 25. rand()
- 26. mt_rand()
- 27. mt_rand()
- 28. 3 / Delivery: make, test, deploy, rollout
- 30. And BOOM the awesome-awesome new feature!!1!
- 31. Feature Toggling Switch ● Traditional turned on/off ● Helps you to keep a feature in master ● Can be used as override Percentage ● Useful for A/B tests ● Minimize load risks Condition ● Very specific ● The “logged in” test ● Split users by clusters
- 33. A word about backend metrics ● Time series based databases ● Carbon (StatsD), Graphite, Grafana ● composer require liuggio/statsd-php-client
- 34. Dashboard: visualize distribution (stacked) legendValue(alias(scale(stacked(stats.web.registration.social.count),60),"Social"),"last") legendValue(alias(scale(stacked(stats.web.registration.master.count),60),"Master"),"last")
- 35. Dashboard: visualize distribution (%) legendValue(alias(stacked( asPercent(stats.web.registration.social.count,sumSeries( stats.web.registration.social.count, stats.web.registration.master.count))), "Master"), "last") legendValue(alias(stacked(asPercent(sumSeries( stats.web.registration.master.count),sumSeries( stats.web.registration.social.count, stats.web.registration.master.count))),"Social"),"last")
- 36. Dashboard: track application responses alias(hitcount(stats.web.registration.response-200,"5minutes"),"Ok") alias(hitcount(stats.web.registration.response-400,"5minutes"),"Bad Request") alias(hitcount(stats.web.registration.response-500,"5minutes"),"Internal Server Error") ...
- 37. 4 / Wrap up all the things!
- 38. Key Learnings ● Track all the things! ● Trust statistics, don’t DYI ● Pay attention to team interactions ● Use a circuit breaker ● Define a fallback strategy ● Allocate cleanup time in advice ● Build packages, split things ● Handle errors properly ● Consider vendors (Mixpanel, Optimizely...)
- 40. Credits ● Riccardo Tortul for the math/stat background ● Pictures a. Photos are my own, except b. http://it.disney.wikia.com/