Advanced A/B Testing - Jax London 2015
Download as PPTX, PDF1 like1,025 views
Aviran Mordo presented on A/B testing and Wix's open source experiment system called Petri. Wix runs hundreds of experiments continuously to test new features on a subset of users before broad release. Petri allows configuration of multivariate tests, automated assignment of users to test groups, and integration with code deployment and issue tracking systems. The talk covered best practices like excluding bots, gradual rollout, and handling experiments across shared documents.
Advanced A/B Testing - Jax London 2015
- 1. Aviran Mordo Head of Engineering @aviranm linkedin.com/in/aviran aviransplace.com Advanced A/B Testing
- 3. Wix In Numbers Over 72M users + 1.5M new users/month Static storage is >2Pb of data 3 data centers + 3 clouds (Google, Amazon,Azure) 2B HTTP requests/day 1000 people work atWix
- 5. BasicA/B testing Experiment driven development PETRI –Wix’s 3rd generation open source experiment system Challenges and best practices Complexities and effect on product Agenda
- 8. Home page results (How many registered)
- 10. This is the Wix editor
- 11. Gallery manager What can we improve?
- 12. Is this better?
- 14. Product ExperimentsToggles & Reporting Infrastructure
- 15. How do you know what is running?
- 16. If I “know” it is better, do I really need to test it? Why so many?
- 18. Sign-up Choose Template Edit site Publish Premium The theory
- 21. EVERY new feature is A/B tested Measure success If flawed, the impact is just for % of our users Conclusion
- 24. New code can have bugs Conversion can drop Usage can drop Unexpected cross test dependencies Sh*t happens (Test could fail)
- 25. Numbers look good but sample size is small We need more data! Expand Reaching statistical significance 25% 50% 75% 100% 75% 50% 25% 0%Control Group (A) Test Group (B)
- 26. Language GEO Browser User-agent OS Minimize affected users (in case of failure) Gradual exposure (percentage of…) Company employees User roles Any other criteria you have (extendable) All users
- 27. First time visitors = Never visited wix.com New registered users = Untainted users Existing registered users = Already familiar with the service Not all users are equal
- 28. Signed-in user Test group is determined by the user ID Guarantee toss consistency across browsers Anonymous user (Home page) Test group is randomly determined Cannot guarantee consistent experience cross browsers 11% ofWix users use more than one desktop browser Keeping consistent UX
- 29. Which Test Group are You? Non-Developers Developers Test Group A Test Group B
- 30. Calling Laboratory is Easy
- 33. Adding a mobile view
- 34. First trial failed Performance had to be improved
- 35. Halting the test results in loss of data. What can we do about it?
- 36. Solution – Pause ! • Maintain NEW experience for already exposed users • No additional users will be exposed to the NEW feature
- 37. Decision (What to do with the data) Keep feature Drop feature Improve code & resume experiment Keep backwards compatibility for exposed users forever? Migrate users to another equivalent feature Drop it all together (users lose data/work)
- 39. Always exclude robots Don’t let Google index a losing page Don’t let bots affect statistics
- 41. # of active experiment Possible # of states 10 1024 20 1,048,576 30 1,073,741,824 Possible states >= 2^(# experiments) Wix has ~600 active experiments ~4.149516e+180
- 42. Integrated into the product
- 44. Share document with other users
- 45. Document owner is part of a test that enables a new video component
- 46. What will the other user experience when editing a shared document ? Owner Friend
- 47. Assignment may be different than owner’s Owner (B) Friend (A)
- 48. Enable features by existing content What will happened when you remove a component Enable features by document owner’s assignment The friend now expects to find the new feature on his own docs Exclude experimental features from shared documents You are not really testing the entire system Possible solutions
- 50. Petri is more than just an A/B test framework Feature toggle A/B Test Personalization Internal testing Continuous deployment Jira integration Experiments Dynamic configuration QA Automated testing
- 51. Petri is an open source project https://github.com/wix/petri
- 52. Q&A Head of Engineering Aviran Mordo @aviranmwww.linkedin.com/in/aviranhttp://www.aviransplace.co m https://github.com/wix/petri http://goo.gl/yMUzta
- 54. Modeled experiment lifecycle Open source (developed usingTDD from day 1) Running at scale on production No deployment necessary Both back-end and front-end experiment Flexible architecture Why Petri
Editor's Notes
- #5: Who here does A/B tests? Who plans to do A/B test?
- #6: A/B test is embedded in our development process Petri is based on our experience and lessons we learned
- #7: You divide your users into group and measure which reached your goal
- #8: What does it mean better? What is your goal?
- #9: Measure conversion to register
- #10: Not just comparing pages but also in development
- #12: The theory – we can make a better gallery Our goal – make it easier for our users to build their sites (converting to premium)
- #14: It is not about winning, its about not losing
- #15: Lessons learned from 5 years of experience Petri allows PM to manage their tests
- #16: A screenshot of the UI we built on top of PETRI
- #18: PM added a Premium link in the editor
- #19: If we shorten the funnel more users will reach the purchase page, thus increasing our sales PM added a Premium link in the editor
- #20: Why did it fail. T-Shirt time
- #21: Users were taken out of editing context before they were happy with the results
- #23: Who thinks we should start with 50%
- #24: Remember a test could fail
- #26: The end result of every A/B test is reaching a decision. For this we need enough numbers. Add %, countries etc’
- #29: For anonymous users – this is the best we can do. This means sometimes (~11%) users will see different experiences.
- #31: Product manager defines a limited new experiment
- #32: Product manager defines a limited new experiment
- #33: We also test new must have features
- #34: There is no A version. Control group just don’t get it.
- #35: we need to improve before releasing to all users.
- #36: Lose mobile view ? Unable to update ?
- #37: Pause is a temporary state until system improves and resume test
- #39: What would you expect the result should be for a bot? A? B? 2-nd T-Shirt time!
- #42: Production is never in a ‘known’ state At least 2^ (more than 2 options)
- #49: The obvious answer may be – allow the friend to edit the component if it’s already in the site But then – what if the friend deletes the component by mistake (or on purpose)? Then if he’s assigned to A he won’t be able to add it back. Possible solution – assign by site owner instead of by user (this means you must implement server side state) (why? Bcos you don’t know what lang/geo etc the site owner was when he got assigned – you only know his user id) Not perfect, user may experience something else on his own document
- #51: Expose features internally to company employees Select assignment by sites (not only by users)
- #54: santa