Advancing Testing Program Maturity in your organization
3 likes872 views
The document outlines an advanced A/B testing program aimed at enhancing testing maturity within organizations. It discusses the key phases of program maturity, strategies for effective testing, and the roles of various team members in the testing workflow. It emphasizes the importance of data-driven decision-making, executive support, and integrating testing into the organizational culture to optimize performance and accountability.
Advancing Testing Program Maturity in your organization
- 1. Intended for Knowledge Sharing only Advancing Testing Program Maturity Optimizely User Group – San Francisco Oct 2017
- 2. Intended for Knowledge Sharing only RAMKUMAR RAVICHANDRAN Intended for Knowledge Sharing only 2 Director, Analytics at Visa, Inc. Data Driven Decision Making via Insights from Analytics, A/B Testing and Research Manager, Analytics & AB Testing Data Driven Decision Making via Insights from Analytics, A/B Testing and Research ROGER CHANG MIMI LE Senior Director Product Launch Management
- 3. Intended for Knowledge Sharing only Quick recap of what it is Intended for Knowledge Sharing only Just venting it out a little… 3
- 4. JUST CHANGE THE COLOR OF THE BUTTON, DAMN IT! Intended for Knowledge Sharing only
- 5. WAIT A MINUTE! WHAT WILL YOU DO WITH THAT MUCH MONEY, SENORE? Intended for Knowledge Sharing only
- 6. YOU BROKE IT, DUDE!!! Intended for Knowledge Sharing only
- 7. THE NUMBERS AREN’T IN LINE – YOU MUST HAVE SCREWED UP! Intended for Knowledge Sharing only
- 8. WHY WOULD I NEED YOU, WHEN I GOT ARTIFICIAL INTELLIGENCE? Intended for Knowledge Sharing only
- 9. Intended for Knowledge Sharing only Quick recap of what it is Intended for Knowledge Sharing only A typical Testing Program in Utopia… 9
- 10. IF GOD DECIDE TO CREATE AN A/B TEST PROGRAM, WHAT WOULD IT LOOK LIKE… Intended for Knowledge Sharing only Every major product change has been iterated, quantified & contextualized A centralized but modular, seamless & integrated Learn, Listen and Test Framework covering all domains A Single-Source-Of-Truth Testing Datamart within the Organization’s Datalake for year end Program effectiveness studies Unified Workflow & Project Management with searchable Knowledge repository & centralized Admin capabilities Programmatic Testing with human intervention protocols
- 11. Intended for Knowledge Sharing only Quick recap of what it is Intended for Knowledge Sharing only What is a Testing Maturity Curve? 11
- 12. TESTING PROGRAM MATURITY CURVE SELL SCALE EXPAND DEEPEN TRANSFORM Phases of Maturity ValueAdd We are here
- 13. DEEPEN • Content Delivery, Personalization, Champion-Challenger Set up • Platform: Cross Integration with Analytics, ML, Session Replays & Research at Application Layer • Predictive Analytics on Test Impact: Test-Mix Models (Scenario Planning & Scoring) • Unified Workflow & Project Management EXPAND • Complexity and scope of the tests • Multi (Variate, Pages, Experience, Device, Domain) • Enterprise Framework (Server Side Integration, Datamart) • Enterprise rollup of Operational and Strategic Impact • Searchable Knowledge bank & Feedback loop into Design Stage TRANSFORM TESTING PROGRAM MATURITY- PHASES PHASES KEY ACTION ITEMS SUCCESS CRITERIA SELL • Buy-ins across leadership & stakeholders • Scrappy quick win tests • Allaying fears of Dev/QA/Security org • Tangible KPI impact • Sponsor Business Units and victory use cases: Prod, UX, Mktg • Approval for a cross functional team and Testing environment set up SCALE • Agile Workflow set up • Test Pipeline created & shared • Testing Dashboard • Readouts shared with stakeholders • A successful rollout because of Test & Learn Initiative (Use Case Driven- >Numbers Driven->Experience Driven) • Testing formalized within Dev Cycle • Algorithmic Test Management (Traffic adjustments, winner ramps, combinatorial tests) • Test Modularity & Portability • Testing as “Monetizable” Product • Test & Learn made self serve via Trainings for Citizen Experimenters • Cross Pollination across BU – within the DNA of the organization
- 14. TRANSFORM TESTING PROGRAM MATURITY- EXPERIENCE & LEARNING PHASES CHALLENGES RESOLUTION THAT WORKED FOR US SELL • Executive Buy-ins • Pushback from Security, Branding, Integration, Development & QA teams • Proof-Of-Concept (guard against weak POC & Sponsor BU) • Risk Ownership with executive air cover/Shared limelight • CMS/Bug Fixes – Good and bad! SCALE • Resourcing & Funding support: Availability & size of shared team • Sandbox availability/sync with release cycles/broken tagging • Production ramp • Show progress even with persisting challenges • Successful Project delivery • Dashboards & Communication readouts EXPAND • Sample size, cookie issues and cross domain traffic. Interaction problems. • Consistency and integration issues of tagging and logic between front and back end and within backends itself • Knowledge Management site and dashboard • Instrumentation request for the Engineering team to link the various cookies and identifiers DEEPEN • Huge investment and potential tradeoffs with re-architecting instrumentation • Resourcing and Funding into platform set-up & product builds • Potentially Test-Mix Models on manually scraped metadata (less rigorous) • Server Side product set up • Dependent on successful transition from previous phase • Resourcing & funding • Test & Learn made self serve via Trainings for Citizen Experimenters. Brown bags, whitepapers?
- 15. Intended for Knowledge Sharing only Quick recap of what it is Intended for Knowledge Sharing only What has worked for us so far… 15
- 16. KNOWING WHAT WE ARE TESTING & HOW MUCH TO EXPECT… Message Prominence Flow Form Clear and crisp Value Prop and Call to Action (CTA) Trendy and easy to spot Easily spotted and fitting with the Consumer’s mental model Navigation and Pathing Minimal and relevant elements only Placement Personalization Personalized with behavioral insights Content Algorithmic delivery/Contextualized Content Performance Platform Performance (Latency, Uptime, Errors) ExpectedImpact Intended for Knowledge Sharing only
- 17. PLANNING IT END-TO-END • Analytics team creates direct/proxy metrics to measure the performance • Instrument metrics if needed • Decision on the Research Methodology based on Analytical findings ACTIONS • Defined the question to be answered and why, Design the changes, know the cost and finalize success criteria • Quantify/Analyze the impact • Size the potential impact on launching Measure LaunchStrategy PHASES Analyze Primary Metrics, e g., • Click Through Rate • NPS Secondary Metrics • Repeat Visits • Lifetime Value Questions • Target Customers • Where and What is being checked? • Why is this even being considered? • Target Metrics and success criteria Research Methods • Attitudinal vs. Behavioral • Qualitative vs. Quantitative • Context for Product Use Factors deciding Research Methods • Speed of execution • Cost of execution • Reliability • Product Development Stage Factors deciding eventual rollout (in order of priority) • Strategic need • Estimated impact calculation from Analytics • Findings from other sources (Data Analytics/Mining, Consumer Feedback DETAILS Intended for Knowledge Sharing only
- 18. KNOWING WHEN TO SET UP A/B TEST AND NOT… Method Description Factors Speed Cost Inference Dev Stage Prototyping Usability Studies Focus Group Surveys & Feedback Pre-Post A/B Testing Create & Test prototypes internally (external, if needed) Standardized Lab experiments – Panel/s of employees/friends/family In-depth interviews for Feedback Email/Pop-ups Surveys Roll-out the changes and then test for impact Different experiences to users and then measure delta Quickest (HTML Prototypes) Quick (Panel, Questions, Read) Slow (+Detailed interviews) Slower (+Response rate) Slower (Dev+QA+ Launch+Release cycle) Slowest (+Sampling+ Profiling+ Statistical Inferencing) Inexpensive (Feedback incentives) Relatively expensive (+Lab) Expensive (+Incentive +Time) Expensive (Infra to send, track & Read) Costly (+Tech resources) Very Costly (+Tech +Analytics +Time) Directional +Consistency across users +additional context on Why? +strength of numbers +Possible Statistical Significance but risk of bad experience. +Rigorous (Statistical Significance). *Risk of bad experience reduced. Ideation Stage Ideation Stage Ideation Stage Ideation/Dev/ Post Launch Post Launch Pre Launch (after Dev) Intended for Knowledge Sharing only
- 19. PAYING YOUR DUES –RIGHT WORKFLOW MANAGEMENT & XFUNCTIONAL OWNERSHIP • A/B Test Analyst (Analytics): The driver of the testing program. Involved from start to finish up until the hand-off of a successful test to its respective product owner. A SME in the Optimizely tool, owner of test setup, deployment, and analysis. • Product Partner: Talks to and brings in the right people for different steps of the process. Offers product’s perspective in terms of gatekeeping duties on test ideas. Well connected to different product owners and acts as the liaison towards the product team. • QA Partner: Helps ensure that there are no bugs in the test setup, from a usability standpoint. • Technology Partner: Offers consultation on feasibility for tests, assists in setup of advanced tests. • Design Partner: Helps the team germinate ideas, as well as give the team visuals to work off of in a test. Ideation Prioritization / Grooming Setup QA Deployment Analysis Implementation Analytics, Product, Design, Tech Analytics, QA Analytics, Product Intended for Knowledge Sharing only
- 20. …AND FOCUS ON FULL SPECTRUM OF OPERATIONAL METRICS Operational Program KPIs • # of Tests run per month • % Successful tests • % Learning Tests • % Workaround/Bug fix Tests • #Channels Tested on • Time from ideation to deployment • Time from test outcome to product implementation • Program RoI • Stakeholder NPS • KPI Delta vs. Universal Control Intended for Knowledge Sharing only …both raw and YoY growth forms
- 21. & ANALYTICS VALUE CHAIN: STRATEGY DRIVES EVERY INITIATIVE & ANALYTICS MEASURES ITS EFFECTIVENESS! Analytics provides insights into “actions”, Research context on “motivations” & Testing helps verify the “tactics” in the field and everything has to be productized… Strategy Data Tagging Data Platform Reporting Analytics Research Cognitive Iterative Loop Key benefits Focus on Big Wins Reduced Wastage Quick Fixes Adaptability Assured execution Learning for future initiatives Intended for Knowledge Sharing only Optimization
- 22. …& TIMING IT CORRECTLY WITHIN THE ANALYTICS MATURITY RAMP Testing makes sense after we know what the baseline actually looks like… Intended for Knowledge Sharing only 60% 20% 10% 5% 5% 20% 30% 15% 10% 5% 20% 25% 25% 25% 20% 25% 25% 20% 15% 25% 20% 20% 20% 20% 15% YEAR 1 YEAR 2 YEAR 3 YEAR 4 YEAR 5 Primary source of insights for Decision Making along the Analytics Maturity Curve Reporting Data Analytics User Research A/B Testing Advanced Analytics/Machine Learning Data Products Cognitive Analytics ILLUSTRATIVE
- 23. MAKE OR BREAK DIMENSION: PROJECT TRACKER & PERFORMANCE DASHBOARD Priority Test Description Requestors/Key Stakeholders Type of Change Hypotheses How did we arrive at this hypotheses Where will the Test happen? Target Audience 1 Remove Ad banner on Yahoo home page User Experience Prominence Removing Ad banners would reduce distraction and focus users to CTA Product/Design Judgement Home Page All Consumers Standard Test Plan Document Ready #Test Cells #Days needed for the Test to run tor statistical significant sample Design Ready? Specific Technical Requirements? Estimated Tech Effort/Cost (USD) Overall Test Cost (USD) Yes 2 40 Yes Test Details Other details from the Test ILLUSTRATIVE
- 24. MAKE OR BREAK DIMENSION: PROJECT TRACKER & PERFORMANCE DASHBOARD CONTD Intended for Knowledge Sharing only ILLUSTRATIVE Primary Metrics Secondary Metrics Estimated Benefit (USD)Click Through Rate Net Promoter Score Repeat Visits Customer Lifetime Value x% y% z% a% Expected Impact from the Test Primary Metrics Secondary Metrics Estimated Benefit (USD)Click Through Rate Net Promoter Score Repeat Visits Customer Lifetime Value x% y% z% a% Actual Impact from the Test
- 25. & COMMUNICATION READOUT AT REGULAR CADENCE! Objective Understand if removing Ad banner on home page improves click through rate on articles and increases consumer satisfaction 0% 20% 40% 60% 80% 100% 120% 0% 2% 4% 6% 8% 10% 12% 14% 16% 18% DeltabetweenTest&Control Test/ControlValues Test metrics - Click through Rate Delta Test Control Key Findings 1. Removing the banner increased CTR by '100%' and NPS by 20 points '. It translates to $40 M in Lifetime Value impact. 2. All the above lifts are statistically significant at 90% confidence level. These lifts were also consistent over two weeks time window. Sl.No. 1 2 3 5 Performance data Time window: Apr 1, 1980 to Apr 14, 1980 ILLUSTRATIVE
- 26. THINGS WE WATCH OUT FOR • Engineering overheads – every time a new flow needs to be introduced or any major addition to the experience, new development is required. It has to go through Standard engineering prioritization route unless a SWAT team is dedicated to it. • Tricky QA situations – QA team should be trained to handle A/B Testing scenarios and use cases; Integration with automated QA tools. Security and FE load failure considerations apart from standard checks. • Operational excellence requirements – Testing of the Tests in Sandbox, Staging and Live Site Testing areas. End to End Dry runs mandatory being launching the tests. • Analytical nuances – Experiment Design supreme need! External factors can easily invalidate A/B Testing. Sample fragmentation with increasing #tests and complexity; Need for Universal Control; Impact should be checked for significance over time. • Data needs – Reliable instrumentation, Testing Tool JavaScript put in right place, with minimal overhead performance impact, integration with Web Analytics tool, Data feed with ability to tie with other data sources (for deep dives). • Branding Guidelines – Don’t overwhelm and confuse users in quest for multiple and complex tests; Standardize but customize experience across various channels and platforms; Soft launches should be as much avoided as possible. • Proactive internal communication, specifically to client facing teams. • Strategic Decisions – Some changes have to go in irrespective of A/B Testing findings, the question would be how to make it happen right? This is gradual ramp, progressive learning and iterative improvements – a collection of A/B Tests and not one off big one. …A/B Testing can never be a failure, by definition it is a learning on whether the change was well received by the user or not that informs the next steps
- 27. Intended for Knowledge Sharing only Quick recap of what it is Intended for Knowledge Sharing only Discussion items 27
- 28. QUESTIONS FOR THE AUDIENCE Where are you in the Testing Maturity Curve? What were your biggest bottlenecks and how did you solve them? Were you successful in up-leveling the conversation in your organization? How did you crack the Resourcing & Funding problem? What are the things that worked best for you in your journey? How did you protect Testing resources from being used up for CMS or Bug Fixes? How did you manage the nuance between Learning and Business Objectives? How did you convince the organization to use Testing as driver of accountability but also not get dragged into for political issues? Intended for Knowledge Sharing only
- 29. Intended for Knowledge Sharing only Quick recap of what it is Intended for Knowledge Sharing only The parting words… 29
- 30. KEY TAKEAWAYS Intended for Knowledge Sharing only An advanced Experimentation program is hallmark of a “Data Driven Decision Making Culture” of accountability & transparency Benefit from Experimentation is best realized when it’s anchored to Strategic goals and is driven with insights from Analytics and Research Mature organization leverage Algorithmic Test Management framework to achieve scalability and efficiency at Optimal Program RoI levels Organizations with a disciplined Experimentation culture within the DNA are poised to reap benefits of higher accountability, focus on business performance and optimized Customer Experience Management Testing Program is a high reward but high investment-high political risk function and an executive leadership & support are imperative
- 31. Intended for Knowledge Sharing only Quick recap of what it is Intended for Knowledge Sharing only Appendix 31
- 32. THANK YOU! Would love to hear from you on any of the following forums… https://twitter.com/decisions_2_0 http://www.slideshare.net/RamkumarRavichandran https://www.youtube.com/channel/UCODSVC0WQws607clv0k8mQA/videos http://www.odbms.org/2015/01/ramkumar-ravichandran-visa/ https://www.linkedin.com/pub/ramkumar-ravichandran/10/545/67a RAMKUMAR RAVICHANDRAN ROGER CHANG https://www.linkedin.com/in/rogervchang/