Mailchimp: Scaling Experimentation Across Teams
2 likes676 views
The document discusses the transition of experimentation from marketing to product development, highlighting the importance of evidence-based optimization in digital experiences. It details Mailchimp's journey in fostering a culture of experimentation, emphasizing the need for cross-team collaboration and continual iteration based on insights gained from experiments. The presentation encourages organizations to leverage data-driven decision-making and provides a framework for implementing successful testing strategies.
Mailchimp: Scaling Experimentation Across Teams
- 1. 1 How Experimentation has Moved from Marketing to Product Anne Reints & Alex Lee +
- 2. 2 Alex Lee Customer Success Manager Anne Reints Senior Manager, Growth Analytics & Experimentation Introductions
- 3. 3 Housekeeping ● We are recording this webinar ● You’ll receive the recording and slides ● We’ll answer all questions at the end
- 5. 5 Agenda ● Introduction ● About Optimizely ● Mailchimp’s Journey ● Q&A
- 6. Revenue, share of wallet, funnel conversion, risk mitigation, ops efficiency What We Do Next gen “Test and Learn” platform for enterprise-wide digital experience optimization Over 1B impressions daily Replaces Digital Guesswork With Evidence-based Optimization Digital Experimentation SaaS Platform Apply the scientific method to “at-scale” business decision making. 20X increase in Yield
- 7. “Our success is a function of how many experiments we do per year, per month, per week, per day.” Jeff Bezos “Our aim is to create the best product for our customers, and we do that through constant innovation and testing.” Gillan Tans, CEO “Our company culture encourages experimentation and free flow of ideas.” Larry Page “We use experimentation and testing to inform as much of the business as we possibly can” - Gregory Peters, CPO Today’s Digital Leaders Win By Using Experimentation At-Scale
- 8. What We Enable 1500+ +51% +$21M +19% +28% Improved subscription acquisition Users Scale and agility of experiments; users incremental in revenue Improved Driver Acquisition increase in cross-sell performance
- 9. Spent with Digital Media 19hr week 2018 Digital-Centric Customer Spent with Digital Media 41 $396m Mobile commerce Mobile commerce $1 8t. 2008 Digital-Convenient Customer hr week Your customer has fundamentally changed how they engage
- 10. Experimentation: Testing variations of digital experiences with real users and deploying what wins +21% -5%
- 11. Frontend UI Backend Business Logic & Data The anatomy of an experience i.e. navigation, search location & visual treatment Copy Images & Colors Layout Search algorithms Personalize content based on previous behavior Recommendations Make your headlines more personal
- 12. We’re Proud to Work With Great Global Enterprises 24 of the Fortune 100 have chosen Optimizely to drive their digital experience
- 13. 2019 Growing Growth at Mailchimp Digital Change Agents Webinar Series Anne Reints Senior Manager, Growth Analytics and Experimentation
- 14. 2019 The History of Growth at Mailchimp
- 15. 2019 The History of Growth at Mailchimp 20182017 2019 July Marketing Optimization is born Nov It’s not just a team of one anymore July Experimentation Resource Group is Born Feb Product Analytics begins Jan Marketing Optimization CFX is born April The Growth Product team is Born June We launch our 1st experiment! Nov We move to Domains/Squads x 30
- 16. 2019 Getting your team on board
- 18. 2019 Invite everyone to help solve the problem
- 19. 2019 Find blockers and optimize together
- 20. 2019 What this looks like in action
- 21. 2019 Create a shared understanding of the problem or opportunity Problem Statement: Once a user uploads 2,000 or more contacts, they are no longer eligible for the free Mailchimp plan. How can we help make this transition easier?
- 22. Users begin on the Dashboard 90% Navigate to Lists (Audiences) page 80% Import Match 25% Import Review 20% Many steps later, users finally land on the Sent Campaign screen 19% Import Setup 50%
- 23. 2019 Sneak peek into our “Innovation Lab”
- 24. 2019 Sneak peek into our “Innovation Lab”
- 26. 2019 Control Variant 1 Variant 2 Hypothesis: If we provide an easier upgrade path through a new modal and a clear CTA, along with more positive language to make this a celebratory moment, we will increase paid user conversion rate. Audience: Users that have updated any draft in the last 30 days and have >2000 contacts and currently do not have billing information stored in Mailchimp
- 27. 2019 Control Variant 1 Variant 2 Results: Variant 2, which contained copy and button updates, was this winner with a 20.4% lift in new paid conversions compared to the control variant. Hypothesis: If we provide an easier upgrade path through a new modal and a clear CTA, along with more positive language to make this a celebratory moment, we will increase paid user conversion rate. Audience: Users that have updated any draft in the last 30 days and have >2000 contacts and currently do not have billing information stored in Mailchimp
- 28. 2019 Analyze results as a team
- 29. 2019 Create a framework for data driven decisions across the company
- 30. 2019 Defining KPIs and your North Star Metric
- 31. Proprietary & Confidential 2019 Strategic Business KPIs Business Health KPIs Total Revenue, Profit Margin, Operating Cost Business Level KPIS Domain Level KPIs Clarifies how we prioritize and drive business value Articulates the customer problems we are trying to solve w/ our North Star Key data points that inform the customer problem North Star Problem Statement KPIs Input metrics KPIs that measure your company’s growth
- 32. Proprietary & Confidential 2019 Strategic Business KPIs Business Health KPIs Total Revenue, Profit Margin, Operating Cost Business Level KPIS Domain Level KPIs Clarifies how we prioritize and drive business value Articulates the customer problems we are trying to solve w/ our North Star Key data points that inform the customer problem North Star Problem Statement KPIs Input metrics KPIs that measure your company’s growth
- 33. Proprietary & Confidential 2018 33 An Example: Email Sign Up Onboarding EngagementCreate a HabitAha Moment Uploads a List Sends first campaign Sends 3 campaigns in first 90 days Sends monthly campaigns + uses other features Sign up
- 34. Proprietary & Confidential 2018 34 An Example: Email Uploads a List Sends first campaign Sends 3 campaigns in first 90 days Sends monthly campaigns + uses other features Validate Sign up What % of sign ups upload a list? Are users who upload a list more likely to reach the Aha Moment? What % of users complete this action? How does this map to our business KPIs? Are users who complete this action more likely to be retained? How does this map to our business KPIs? What other factors deepen engagement and help retain users? Total # of sign ups (Top of funnel) Sign Up Onboarding EngagementCreate a HabitAha Moment
- 35. Strategy Lift Required Impact on Paid Users Drive Engagement 500% 30% Improve Activation 25% Less Effort 30% More Impact Look for smaller efforts and bigger impact
- 36. 2019 The customer problem you are working to solve North Star Metric How your company makes money Business level KPIs
- 37. 2019 It measures the moment a customer finds value in your product It represents the core of your domain’s strategy It can be measured immediately after launching a feature or creating a new experience It correlates with business KPIs that measure Mailchimp’s success Qualities of a North Star Metric
- 38. 2019 It measures the moment a customer finds value in your product It represents the core of your domain’s strategy It can be measured immediately after launching a feature or creating a new experience It correlates with business KPIs that measure Mailchimp’s success Qualities of a North Star Metric
- 39. 2019 It measures the moment a customer finds value in your product It represents the core of your domain’s strategy It can be measured immediately after launching a feature or creating a new experience It correlates with business KPIs that measure Mailchimp’s success Qualities of a North Star Metric
- 40. 2019 It measures the moment a customer finds value in your product It represents the core of your domain’s strategy It can be measured immediately after launching a feature or creating a new experience It correlates with business KPIs that measure your company’s success Qualities of a North Star Metric
- 41. 2019 Use data and insights to evaluate where you have the most opportunity
- 42. 2019 Dig into your input metrics with both qualitative and quantitative research
- 43. 2019 Make a statement about how you think you’ll impact your metrics, and be able to measure that once you release
- 44. 2019 When you see the results, be humble— we’re often wrong about the solution the first time.
- 45. 2019 An Example: Themed Templates Hypothesis: By surfacing themed templates in the default template selection window, we can give users a head start and content, helping them to activate more quickly.
- 46. 2019 An Example: Themed Templates Hypothesis: By surfacing themed templates in the default template selection window, we can give users a head start and content, helping them to activate more quickly.
- 47. 2019 Once we know we’re wrong, dig in with more research
- 48. 2019 Understand why and let these new insights drive your future roadmap
- 49. 2019 Creating a Culture of Experimentation and Inviting Others to Join You
- 50. 2019 Engage others in the company and make them a part of the process. Share your learnings. Tell a story with your experiments and help your company learn. Share your impact. As your experiments lead to impact on the business, show what impact you’ve had and how experimentation helped you get there. STEP 1: Tell everyone what you’re doing
- 51. 2019 Control Variant 1 Variant 2 Results: Variant 2, which contained copy and button updates, was this winner with a 20.4% lift in new paid conversions compared to the control variant. Hypothesis: If we provide an easier upgrade path through a new modal and a clear CTA, along with more positive language to make this a celebratory moment, we will increase paid user conversion rate. Audience: Users that have updated any draft in the last 30 days and have >2000 contacts and currently do not have billing information stored in Mailchimp
- 52. 2019 We created Experimentation, Ideation, Analytics, and Developer trainings and shared them with other product teams. Provided consultations to teams getting started. These were great opportunities for everyone on our cross functional teams to get involved and grow the number of subject matter experts in the organization. STEP 2: Teach them how to do it too
- 53. 2019 Invite all teams to share the experiments they’re launching and the results (we do this using our company messaging app). When a team is getting started - celebrate with them. Even when they lose, help them celebrate the learning. At some point, with team’s that are new to experimentation (and not so new) things will go terribly wrong - make sure you celebrate every attempt as a success in terms of learning, even when it’s painful. STEP 3: Celebrate their success
- 54. 2019 Changing your culture will never happen as quickly as you want it to, and not everything you do will work. Be open to changing your process, how you’re communicating, and empower teams to take ownership of what they’re learning and change the process with you. Free food is always a great motivator for getting others involved. STEP 4: Keep iterating and applying your learnings
- 55. 2019 Q&A
- 56. 2019 Thank you.