Opticon 2017 Advanced Program Management
Download as PPTX, PDF1 like674 views
A successful experimentation program involves assessing opportunities, prioritizing experiments, and measuring program results. It assesses potential experiments by calculating expected value based on win rate, average lift, and revenue impact. Experiments are prioritized using a scoring model addressing business strategy, customer impact, opportunity size, and implementation effort. The program is measured using metrics like testing velocity, win rate, and revenue impact to track progress and identify improvement opportunities. Regular assessment allows the program to optimize operations and maximize overall expected value.
Opticon 2017 Advanced Program Management
- 1. opticon2017 What Does a Successful Experimentation Program Look Like? Beth Foster Principal Product Planner, Experimentation, Microsoft
- 6. opticon2017 “The view” - Fort Casey
- 8. opticon2017 Why do you experiment?
- 9. Test my new icon. Tell me which video carousel works best. Find out if mobile customers like our new menus! https://flic.kr/p/6rKxaH
- 10. opticon2017 1. Improve the customer experience 2. Optimize conversion events 3. Validate new features 4. Build an experimentation culture Program goals
- 11. opticon2017 Framework for Program Success Assess opportunities Prioritize backlog Measure program $
- 12. Prioritize backlog • Business strategy • Customer impact • Opportunity • Level of effort to implement https://flic.kr/p/7VFBK2
- 13. opticon2017 Prioritize backlog Use an objective, consistent model to score experiment ideas from your teams. Does this align to the core initiatives? Business strategy Key business priority Committed engineering work Will this impact a customers broadly? Customer impact Scope of update worldwide Potential to impact multiple categories and platforms Direct response to customer pain point Is the impact to business high? Opportunity Estimated amount of impact to revenue or conversion activity Impact to overall category How much time does it take to build or plan? Effort Amount of estimated development time
- 14. opticon2017 Prioritize backlog Test ideas submitted Cross-group team approves concepts Experimentation team scores test Prioritization model score added to request
- 15. Measure progress # of tests launched % of traffic tested % of tests deployed globally Test win rate Revenue impact
- 16. opticon2017 Measure progress Track key program measurements to track month over month business progress. Metric Description Monthly Target Actual YTD # of tests launched Measures test velocity 5 4 12 % of traffic tested Tracks testing expected traffic volumes 1M MUV .5 4M % of tests globally deployed % of campaigns tested outside US 50% 50% 40% % of INTL tests % of total volume 60% 50% 55% Test win rate Specific to Optimization tests 30% 40% 35% Revenue impact Estimated annual impact to revenue 5M 6M 12M Example scorecard
- 17. Assess opportunities Define potential impact Set targets Reevaluate quarterly
- 18. opticon2017 Assess Opportunity Home Explore Marketing CLEs Download Details pages Buy PDP pages How-to Help pages 1 Identify conversion by page or page groups 2 Calculate estimated growth 3 Measure overall estimated impact Use to define realistic targets that moves a programs from what’s been done, to what’s possible. 4 Set targets Home PDP Pages Home
- 19. opticon2017 Framework for Program Success Assess opportunities Prioritize backlog Measure program $
- 20. “Our goals can only be reached through a vehicle of a plan” - Pablo Picasso
- 21. opticon2017 What Does a Successful Experimentation Program Look Like? Claire Vo Sr. Director of Product, Optimizely
- 22. opticon2017opticon2017 Building Successful Programs with Optimizely Program Management
- 24. opticon2017 Framework for Program Success Assess opportunities Prioritize backlog Measure program
- 25. opticon2017 PRIORITIZE BACKLOG Capture Ideas Always start with a hypothesis – give teams examples of what a good one looks like. Give structure to testing backlog by categorizing ideas by site / page Naming conventions can help keep things organized and easy to report (Beth is an expert!)
- 26. opticon2017 PRIORITIZE BACKLOG Scoring Potential of the experiment to positively impact your goal Impact to the business is the experiment does win Level of Effort to implement the experiment in Optimizely Love Strategic importance, executive support, biased opinions
- 27. opticon2017 PRIORITIZE BACKLOG Selecting & Accepting Ideas Active status lets everyone know you’ve selected this for experimentation and are beginning work Backlog status lets your team know it’s still under review by the central group
- 28. opticon2017 Democratize (good) ideation • Train teams on how to write a hypothesis • Create a process (like Microsoft’s) for submission, scoring, and acceptance • Give constructive feedback and let teammates share in successes
- 29. opticon2017 Framework for Program Success Assess opportunities Prioritize backlog Measure program
- 30. opticon2017 MEASURE PROGRAM Velocity & Win Rate Testing Velocity lets you measure your teams operational performance Win Rate helps teams measure the quality of their hypotheses and the impact to the business
- 31. opticon2017 Using Program Reporting Right Don’t just look at these reports! • Set monthly, quarterly, and annual goals for performance • Measure and report on a monthly basis • Reassess on an annual basis • Integrated the learnings into the operations of your experimentation program
- 32. opticon2017 ASSESS OPPORTUNITIES Improving Operational Performance Filter your program reporting to uncover interesting insights
- 33. opticon2017 Insights on your program Some things you may learn: • Some teams are more effective than others in moving an idea to experiment quickly • Win rates may be significantly lower on certain pages of your site • Testing velocity may slow around holidays • Certain experiment strategies work more effectively
- 34. opticon2017 You can’t improve if you don’t measure.
- 35. opticon2017 Framework for Program Success Assess opportunities Prioritize backlog Measure program
- 36. opticon2017 ASSESS OPPORTUNITIES Expected Value $1 per roll of the die Every time you roll a 3 I pay you $5 If you want to play this game, come see me after this presentation! Win Rate – 1/6 = 16% Value of Win = $5 Expected Value of Roll = 83 Cents
- 37. opticon2017 ASSESS OPPORTUNITIES Expected Value of an Experiment Win Rate x Average Lift of Winning Test x Revenue Value of Test
- 38. opticon2017 ASSESS OPPORTUNITIES Expected Value of an Experiment Every time you win, you get 5% increase on your revenue, which is $10,000,000 and you win 10% of the time IF IT COSTS LESS THAN $50K TO RUN A TEST, RUN IT! Win Rate = 10% Value of Win = 5% x $10,000,000 = $500,000 Expected Value of Test - $50,000
- 39. opticon2017 ASSESS OPPORTUNITIES Expected Value of an Experiment Expected Value of Experiment x Annual Testing Velocity
- 40. opticon2017 ASSESS OPPORTUNITIES Annual Expected Value of Program High Complexity Low Velocity Low Complexity High Velocity Type of Tests Take longer, bigger changes Easier, smaller changes Win Rate 20% 10% Avg Lift 30% 10% Expected Value of Test $300,000 $100,000 Tests / Year 25 100 Annual Expected Value of Testing Program $15M $10M
- 41. opticon2017 Assessing Opportunity As you begin to understand your program: • Track how win rate and velocity impact your overall program value • Think about ROI and where you can maximize • Take a “portfolio” approach to your experimentation program
- 42. opticon2017 Framework for Program Success Assess opportunities Prioritize backlog Measure program
- 43. opticon2017 Q&A
- 45. opticon2017
Editor's Notes
- #8: Because I know why my family and friends want to travel, I know what to look for – I know how to measure success. So, let’s begin.
- #9: Customer experience improvements, Conversion Optimization, Validation of new features? Measuring program success is not about measuring how many tests you did. You first need to start with, why your program exists, what function is it serving. At Microsoft, we have teams across the company who test. Why they test is different, but at the foundation – all of the teams are working to deliver customer insights and direction to different business groups across the company. So, before we begin, you first have to decide, why your program exists?
- #10: https://www.flickr.com/photos/jsloss/3573190903/ And, if you aren’t careful, to figure out why your are testing, you can quickly find your self at a music festival – where everyone wants something different from you!
- #11: All of the program goals tie back to delivering customer insights. This is not
- #13: https://flic.kr/p/7VFBK2
- #14: A prioritization model creates transparency and focus within testing programs. Ours is a point based model where we score each category (which has a few questions/criteria) and then we total the points. The points map to priority levels which we add to our intake forms, which creates transparency for the overall score.
- #15: A prioritization model creates transparency and focus within testing programs. Ours is a point based model where we score each category (which has a few questions/criteria) and then we total the points. The points map to priority levels which we add to our intake forms, which creates transparency for the overall score.
- #16: https://www.amazon.com/Camping-Campers-Journal-Natural-Brown/dp/1892033011/ref=sr_1_1?ie=UTF8&qid=1507688717&sr=8-1&keywords=camping+log+book
- #17: When measuring, advise is to keep is simple. Look at how you are tracking monthly, and set targets. Some other ideas to track are: bust rate, implementation rate, time to launch, % of inconclusive tests.
- #18: https://www.amazon.com/Camping-Campers-Journal-Natural-Brown/dp/1892033011/ref=sr_1_1?ie=UTF8&qid=1507688717&sr=8-1&keywords=camping+log+book
- #21: Thank you – and here’s to structuring, building and reporting on your Successful experimentation program.
- #35: Customer experience improvements, Conversion Optimization, Validation of new features? Measuring program success is not about measuring how many tests you did. You first need to start with, why your program exists, what function is it serving. At Microsoft, we have teams across the company who test. Why they test is different, but at the foundation – all of the teams are working to deliver customer insights and direction to different business groups across the company. So, before we begin, you first have to decide, why your program exists? >>>> Our goals can only be reached through a vehicle of a plan; Pablo Picasso