From iOS to TiVo: In-app Digital Experience Testing
Download as PPTX, PDF0 likes246 views
The document outlines the evolution of Nike's experimentation culture, emphasizing the adoption of data-driven decision-making to enhance digital products and improve customer experiences. It highlights the success of A/B testing and the use of tools like Optimizely X to increase testing volumes significantly, showcasing various test results that led to improved conversion rates and user engagement. Additionally, it discusses the importance of cross-team collaboration and cultivating a data-driven culture within the organization to meet key objectives.
![23
Optimizely Teams + Slack
Slack integration for
Optimizely Teams & Program
Management is an easy
method to make intake of
ideas easy across the
company.
/Optimizely.idea [Idea Name]
Feature Tests vs. A/B
There are a lot of benefits to
planning your Mobile Apps
in a modular way – using
Optimizely Features.
All Mobile App teams are
embracing Feature Tests.
Tell a Story
Don’t hide behind statistics
and data points.
Testing is about validating a
hypothesis & learning.
Share a valuable, insightful
story. Simplify.
Use the thumb.
Growing up – culture tips & tricks
During this maturation process, we recognized a few insights to “growing up”.](https://image.slidesharecdn.com/closethursfromiostotivofinal-191001172213/85/From-iOS-to-TiVo-In-app-Digital-Experience-Testing-23-320.jpg)
From iOS to TiVo: In-app Digital Experience Testing
- 1. Congratulations to our Outperform Award Winners! Most Dramatic Business Impact Most Customer- Obsessed Company Culture Most Transformative Innovation Most Inspiring Social Impact
- 2. 2 Welcome to Day 2 of Opticon! Sessions today will be recorded and will be available after Opticon Share your feedback and rate sessions on the mobile app
- 3. Will Close Director of Digital Experiments, Nike Alex Alwan Sr. Product Manager Optimization, Fox Networks From iOS to TiVo In-App Digital experience testing
- 4. Will Close Director of Digital Experiments, Nike Running Digital Experiments at Nike
- 5. 5 At Nike: Making Sport a Daily Habit Iconic, inspirational shoe & sportswear brand. Experimentation and athlete feedback has always been critical to Nike’s product development process (and still is today with LeBron, KD...) Founder Bill Bowerman was the Men’s Track Coach of the University of Oregon. He experimented with the design of footwear for athletes – building shoes to the exact specifications of his runners.
- 6. 6 2015: Will joined Nike, to start a “Test & Learn” culture Sports brand, with a culture of winning. One of the world’s strongest brands. Strong intuition (art) has led decision-making at Nike for years. With unprecedented success, globally. Applying consumer data (science) was a new concept to digital product owners. Analytics systems in particular were still immature. Custom in-house, JavaScript testing solution on Nike.com. 4 person team was dedicated to running Nike.com tests only. With a technology background (Microsoft, Xbox, Dell) and specifically in product management, web development, Analytics and A/B Testing.
- 7. 7 Mobile Cart: Primary button (Checkout) is out of view Move Checkout button to top & make it “sticky”
- 8. 8 Mobile Cart: Primary button (Checkout) is out of view Move Checkout button to top & make it “sticky” % to Checkout = +5.5% lift Multi-item Conversion = +5.67% lift
- 9. 9 In Checkout: Using a simple consumer-value proposition Membership value proposition was lengthy marketing-speak. In Checkout, most consumers want to quickly Place their Order, not read. Selection of Membership Checkbox = +172% lift New NIKE Members = +30.3% lift
- 10. 10 Mobile PDP: How to best show multiple related products? Static 2x2 layout vs. carousel vs. 2x2 layout with pagination. Related Usage = +14.4% lift Revenues from Related = +97% lift
- 11. 11 Small, centralized team executing tests 4 team members. Ran about 80-90 tests a year. Technically constrained, front-end only. Lots of low-hanging fruit. Examples showed big impact. Using a home-grown Nike.com testing tool, we showed huge impacts from very small UX changes and leadership recognized the value of A/B testing.
- 12. “ ” macklemore The greats weren’t GREAT because at birth they could paint. The greats were great because they PAINT A LOT.
- 13. 13 2018: Nike deploys Optimizely X Full Stack to scale To enable experimentation from any connected, digital experience – Nike adopts the Full Stack capability across the enterprise.
- 14. 14 Enable Testing by All Product Teams Critical to Product Planning Informs key roadmap priorities Launch + product KPIs increase Features & experiments Create 3-4 options, monitor results All tests run by 1 small, central team Behavior tagging = one of many goals Engagement data = nice-to-have Success = the product launches We launch = Features only We prototype & design = 1 version Before Today
- 15. 1515 No technical limits Support for Client-side & Server-side experiments Built for performance Easy to integrate Lots of example code to leverage. Developer documentation was easy to follow. Integration was quick. Increase Test volume 16 Digital experience squads are integrated & running experiments. Concurrent Test volume has grown 7x – from approx. 5 to 35. Changing norms Growing organic interest in “Test & Learn” capability. Leadership is requiring all teams to experiment. North-star goal: All new UX changes in Nike Digital experiences are rolled out via Feature Tests. Test from anywhere Benefits to-date from federating A/B Test execution to all Nike digital experience squads, via Optimizely X Full Stack.
- 16. 16 Recent Test Results (1 of 3) Subhead
- 17. 17 Recent Test Results (1 of 3) Subhead
- 18. 18
- 19. 19
- 20. 20
- 21. 21
- 22. 22 Visibility is key Creating visibility across the entire Nike program is helpful for leadership. Many different teams have different ways to work. Standardizing where appropriate creates efficiencies. Communicate in multiple formats & cadences (Email, Meetings, Slack..) Bold ideas matter Test features that are going to be released anyway. Good or Bad, Bold ideas help illustrate a learning – and get the audience’s attention (story). Engage your Design & UX teams early and often – to take more risk. Build a community Enable a broad group of “Test & Learn” teammates. Collaborate frequently with this group & make sure they are supported. Use the Voting system for Social feedback on tests. Give them the Stage & Forum to Share. Growing up – culture insights During this maturation process, we recognized a few insights to “growing up”.
- 23. 23 Optimizely Teams + Slack Slack integration for Optimizely Teams & Program Management is an easy method to make intake of ideas easy across the company. /Optimizely.idea [Idea Name] Feature Tests vs. A/B There are a lot of benefits to planning your Mobile Apps in a modular way – using Optimizely Features. All Mobile App teams are embracing Feature Tests. Tell a Story Don’t hide behind statistics and data points. Testing is about validating a hypothesis & learning. Share a valuable, insightful story. Simplify. Use the thumb. Growing up – culture tips & tricks During this maturation process, we recognized a few insights to “growing up”.
- 24. Alex Alwan Sr. Product Manager, Optimization Running Digital Experiments at Fox
- 26. Alex Alwan Sr. Product Manager, Product Strategy Product Management is an art Experimentation is not a choice Experimentation is a tactic not a goal
- 27. Optimization Team From 1 to 2 to 3 to 6 to 4 to 2 people From “Let’s give AB testing a try” to “Why the heck we didn’t AB test that?” From “dunno man, that looks complicated” to “Just follow these steps”
- 28. Experimentation Purposes Improvement Testing Discovery Testing Feature Testing Great to achieve OKRs ”Splitting sign-up and payment screens will improve conversion rate” Great to understand what’s going on, then improve “What would happen if we remove Shows Favoriting from onboarding?” Great to validate/measure features’ impact “What’s the impact of live streaming on VOD viewership?”
- 29. Scaling Is Challenging It’s hard to change people New Skills = New Responsibilities http://millhoppervet.com Someone: “Why AB test? Just launch and see what happens” Me: Use success stories Convince the leaders first
- 30. Product Owner vs. Experiment Owner Responsible Accountable Consulted Informed Scaling Is Challenging
- 31. Scaling Is Challenging “But.. it’s nice and warm inside the box” Optimization is a tactic not a goal Focus on producing learnings not only results pbs.org
- 32. Scaling Is Challenging When there’s no big goal to inspire ipwatchdog.com You see the light at the end of the tunnel? Optimization should serve the company’s OKRs
- 33. Scaling Tips Start small but ask big questions Quantity first, quality later Integrate Data Tracking and A/B Testing systems Center Of Excellence is NOT a gatekeeper
- 34. From Web to Mobile/tvOS Web: Extensions Full Stack: Variables High Flexibility Specific Functionality
- 35. From Web to Mobile/tvOS More Content Starts From Redesigned Collection +31%
- 36. From Web to Mobile/tvOS Add-To-Favorite Rate +15% Push Opt-in Rate +60%
- 37. Tips For Mobile/tvOS Experimentation Create a useless variable for testing purposes Focus on generic capabilities to re-use Don’t break the app, build feature flags
- 38. Feature Flags Takes one out of three possible values: “ON”: Display feature/design to all users “OFF: Don’t display feature/design at all “OPTIMIZELY”: Wait for Optimizely's decision If Invalid, fall-back to “OFF”
- 39. What’s Next? Optimization As Part Of Product Strategy
- 40. Company OKR Product OKRProduct OKR Kye Result Kye Result Kye Result Kye Result Initiative Initiative Initiative Initiative Initiative Hypotheses Hypotheses Where do we need to go?
- 41. Company OKR Product OKRProduct OKR Key Result Key Result Key Result Key Result Initiative Initiative Initiative Initiative Initiative Hypotheses Hypotheses How do we know we’re getting there?
- 42. Company OKR Product OKRProduct OKR Key Result Key Result Key Result Key Result Initiative Initiative Initiative Initiative Initiative Experiments Experiments What will we do to get there?
- 43. Company OKR Product OKRProduct OKR Key Result Key Result Key Result Key Result Initiative Initiative Initiative Initiative Initiative Experiments ExperimentsExperiments Experiments How can we move fast?
- 44. From OKRs to Initiatives, and beyond!
- 45. How to define and plan initiatives? Find Areas of Improvement Design Variations Create KPIs Reports Define Hypotheses Plan Features as Experiments Track Key Results
- 46. Create KPIs Reports 2% Visit -> Subscribe 3 Months 90% Tipping Point 10% Trial Cancelation Visitor to Subscriber Funnel By Day/Hour Visitor to Subscriber Funnel By User Traits Key Result Quality Metrics Deeper Analysis
- 47. Find Areas of Improvement Traffic source User Traits User Behavior Drops and Opportunities Referral traffic from politics news on foxnews.com are 60% more likely to convert Safari Mobile users drop by 85% on funnel’s 2nd step Users who watch on >1 platform are 90% less likely to cancel subscription
- 48. Define Hypotheses User Behavior Users who watch on >1 platform are 90% less likely to cancel subscription Highlight available platforms on the marketing landing page Educate users on other available platforms through email/push messages Display banner ads Fix iOS deep links
- 49. Find Areas of Improvement Design Variations Create KPIs Reports Define Hypotheses Plan Features as Experiments Track Key Results Achieve Growth Hit OKRs Cultivate a data-driven culture Empower product managers to own their OKRs Improve efficiency Transform features into experiments Goals
- 50. Q&A
- 51. Thank you!
- 52. Head to the Expo Hall for a quick break and say hello to our sponsors Up next...learn how to measure experiment impact with Optimizely's Co-founder Pete Koomen!
Editor's Notes
- #27: I started my career in product management with a focus on shipping code and delivering roadmap features, what I found with time I can expand my focus to the success of the product, first thing I looked at was the impact of every release on the product’s main KPIs, things led to others and ended up with fox with a main focus of running ab tests and scale experimentation I also believe that products can’t grow without experimentation, it simply goes back to a simple and fundamental problem solving method: trial and error, which is characterized by repeated attempts which are continued until success. I also believe that we should be careful not to be trapped in experimentations, number of experiments per month or year is a good success metric but should be temporary maybe in the early stages of the optimization program, but focus should shift towards measuring the impact of the product OKRs
- #28: The number of the optimization team members varied with time, went from …….. To…, we experimented multiple strategies including hiring engineers that would build experimentation capabilities, what we found later is that in order to scale optimization we need to empower and enable all teams to run experiments, decentralize optimization so every squad can build, run and measure ab tests It’s been a long journey since we started …
- #30: Scaling is challenging because it has to do with changing the work process which means it will change people’s habits Experimentation is a scientific approach that’s not easy to understand People are smart to understand why we should ab test but sometimes they just need to see it working
- #31: So a new experiment idea came say from Marketing or an executive, what’s next? A brief? Who should create that?