Managing Experiment at Spotify
130 likes16,918 views
Spotify uses metric alignment and A/B testing to optimize user experience across their many autonomous teams and platforms. Education and automation are key to ensuring widespread adoption of the aligned metrics. Presenting results from A/B tests can demonstrate how certain features and functions impact vital metrics like daily and monthly active users, and teams are organized as a people as a service model to best implement optimizations.
Managing Experiment at Spotify
- 1. May 20, 2014 Managing experiments at Spotify Ali Sarrafi Product Owner A/B testing and Experimentation
- 6. May 20, 2014 The right music for every moment 6
- 7. May 20, 2014 7 24M active users. 6M paying subscribers. 20Msongs.
- 8. Tribes Provide fast and reliable access to all the world's music Enable high product development speed while maintaining a highly available service P P P P Squad Squad Squad Squad Chapter Chapter P P P P Squad Squad Squad Squad Chapter Chapter Our team is growing rapidly 250 30 100 10
- 9. 9
- 10. 10 Autonomous fast moving teams Feature 1 Feature 3 Feature 2 Feature 4 Feature 5
- 11. May 20, 2014 Some Statistics Over 40 autonomous teams working on features Over 7 platforms with multiple features Over 3000 source repositories
- 12. 12 We don’t want a Frankenstein software
- 13. 13 Well at least a good looking 2014 edition
- 14. Section name 14 How can we align teams’ optimisation to avoid the frankenstein effect?
- 15. 15 Align on metrics rather than actual tests
- 16. 16 DAU (Daily Active Users) MAU (Monthly Active Users)
- 17. Section name 17 Evolving Metrics
- 18. May 20, 2014 Focus on functionality as a metric
- 19. 19 DAU/MAU Function 1 Function 2 Function 3 Function 4 CTR/Team Metrics Importance (High to low) Order of optimisation
- 20. Section name 20 Now we know what would work how do we make sure everyone uses it?
- 21. 21 It’s all about automation
- 22. 22 and of course education
- 23. Section name 23
- 24. 24 Make magic happen with desired metrics Vital Sign Group A Group B Functionality 1 +1.5% -0.5% Functionality 2 +2.5% -3% Functionality 3 +0.5% 0% Functionality 4 +1% -7%
- 25. 25 People/Infrastructure as a service Feature 1 Feature 5 Feature 4 Feature 2 Feature 3
- 26. Summary • Metric alignment is the most effective way of making sure the overall experience does not suffer • Education and Automation are key for adoption • People as a service is a most effective way of reaching out to teams.
- 27. May 20, 2014 Copyright stuff Napster pic by bixentro, cc 2.0 http://www.flickr.com/photos/bixentro/ Busy record store pic by gomattolson http://www.flickr.com/photos/gomattolson Empty record store pic by passetti http://www.flickr.com/photos/passetti/ First Frankenstein screenshot form : “Frankenstein” movie by Kenneth Branagh, 1994 Second Frankenstein screenshot form : “I, Frankenstein” movie by Stuart Beattie, 2014