Virtual Optimizely Experience 2014 - Kyle Rush - Lessons for Delivering your Best Digital Experience

Editor's Notes

• #22: *These build on 4 separate clicks* Working - employees spend 28% of their days checking and responding to email (Bain Capital) Shopping - e-commerce industry is projected to grow at a rate of 10% YoY - 45% growth by 2018 (Forrester Research) 48% of Facebook’s (1.32 BILLION) users log in on a daily basis (Facebook Earnings Report) 111 Million Americans check daily news online - that’s 54% of Americans who regularly use the internet (Nielsen) Retail (Forrester report cited): http://mashable.com/2013/03/12/forrester-u-s-ecommerce-forecast-2017/
• #23: *These build on 4 separate clicks* Working - employees spend 28% of their days checking and responding to email (Bain Capital) Shopping - e-commerce industry is projected to grow at a rate of 10% YoY - 45% growth by 2018 (Forrester Research) 48% of Facebook’s (1.32 BILLION) users log in on a daily basis (Facebook Earnings Report) 111 Million Americans check daily news online - that’s 54% of Americans who regularly use the internet (Nielsen) Retail (Forrester report cited): http://mashable.com/2013/03/12/forrester-u-s-ecommerce-forecast-2017/
• #24: *These build on 4 separate clicks* Working - employees spend 28% of their days checking and responding to email (Bain Capital) Shopping - e-commerce industry is projected to grow at a rate of 10% YoY - 45% growth by 2018 (Forrester Research) 48% of Facebook’s (1.32 BILLION) users log in on a daily basis (Facebook Earnings Report) 111 Million Americans check daily news online - that’s 54% of Americans who regularly use the internet (Nielsen) Retail (Forrester report cited): http://mashable.com/2013/03/12/forrester-u-s-ecommerce-forecast-2017/
• #25: *These build on 4 separate clicks* Working - employees spend 28% of their days checking and responding to email (Bain Capital) Shopping - e-commerce industry is projected to grow at a rate of 10% YoY - 45% growth by 2018 (Forrester Research) 48% of Facebook’s (1.32 BILLION) users log in on a daily basis (Facebook Earnings Report) 111 Million Americans check daily news online - that’s 54% of Americans who regularly use the internet (Nielsen) Retail (Forrester report cited): http://mashable.com/2013/03/12/forrester-u-s-ecommerce-forecast-2017/ Retail (Forrester report cited): http://mashable.com/2013/03/12/forrester-u-s-ecommerce-forecast-2017/
• #26: *These build on 4 separate clicks* Working - employees spend 28% of their days checking and responding to email (Bain Capital) Shopping - e-commerce industry is projected to grow at a rate of 10% YoY - 45% growth by 2018 (Forrester Research) 48% of Facebook’s (1.32 BILLION) users log in on a daily basis (Facebook Earnings Report) 111 Million Americans check daily news online - that’s 54% of Americans who regularly use the internet (Nielsen) Retail (Forrester report cited): http://mashable.com/2013/03/12/forrester-u-s-ecommerce-forecast-2017/
• #27: Average human attention span: 8 seconds 17% of page views last less than 4 seconds Is your traffic arriving at your website through funnels receiving targeted messaging? Are their experiences targeted, personalized to campaign source? Are you trending towards a more personalized experience overall?
• #29: By using A/B testing to optimize your channels, you will: Improve the efficiency of the channels you use to acquire and retain your customers Prioritize and reinvest in your top-performing channels Generate data in the form of experiments that you can immediately act upon to improve the outcomes for your business. On the right, we have an example of how this lets you do more with the traffic you have. The blue line shows customers acquired per month when a conversion rate holds constant, with a traffic spike in one month. The green line shows steady traffic, but growth in conversion rate. Which scenario leads to more predictable growth, and more customers and revenue?
• #30: Bleacher Report: “The ESPN of the Internet.” Meet Peter, a frontend developer for the team. As a media company, building a loyal readership is a top priority for them. Historically, most of their traffic came from organic search. In 2013, they decided it was time to focus on social media as a new traffic acquisition channel, and try to optimize for social followers. Their hypothesis was that social media traffic would become a more sustainable, loyal readership for their site and app over time.
• #31: Today, the fly-in social modal is the variation that is currently running. What Peter and his team arrived at was a 764% increase in social signups through testing. Not only was this a huge numerical win, but Peter says he feels much more confident asserting his opinion when he can back up his hypotheses with tests. Getting to the current social sharing functionality was an iterative process, and running tests informed every step of that process. His next steps are to expand Bleacher Report’s testing culture, and empower other team members to test their ideas and collect data, too.
• #32: As you can see, their social conversions began to multiply once the team implemented testing (spikes are due to weekly cadence of readership and events in sports. February spike is the Superbowl).
• #35: You have a high-leverage option to make the most out of your marketing spend ... that won’t waste additional budget
• #37: Liftopia is the number one bookings website for lift tickets, as their name so aptly describes. Their website connects internet searchers and growers with ski resorts across the US and internationally - much of their product is focused on search functionality and dynamic price scheduling for the resorts.
• #38: Liftopia evaluates their online channels in the following manner: Visitors x Average Order Size x Conversion Percentage = Revenue. The more they can improve the conversion rate on each channel, the greater their revenue will be. For Liftopia, paid marketing generates a significant part of their site traffic and they use landing pages for traffic from individual paid campaigns. They bid on keywords that express intent, like a combination of a ski resort name and the words “ski deal”.
• #39: Originally, these landing pages included lots of cross-sell and search functionality - other resorts in the area, pictures, lots of places for visitors to get distracted. Hypothesis: Dave and his team felt that the traffic coming from paid search terms was highly qualified traffic, with stronger intent than other traffic channels. They thought that tailoring landing page content with search queries will increase purchase conversions and decrease bounce rates. They removed the search functionality and breadcrumb, and instead presented only the deals for the keyword resort. The result was an improvement of more than 20% in conversions on this page. Since SEM is one of Liftopia’s
• #40: Since this worked so well on the first page, they tested a similar test on the next step of their product funnel. Their date chooser had similar cross-sell functionality to the landing page before it, so they ran the elimination test targeted at the same SEM traffic. They achieved a similar lift - a 23.7% improvement to the second page of their funnel.
• #42: However, optimization isn’t always rosy - here’s another example of a test where taking advantage of a traffic optimizing a content channel did not end well. IGN hoped to increase interest in its videos, a major source of advertising revenue, by making videos more prominent in their navigation. They tested the variation against the existing navigation and found that video conversions plummeted by 93.2%. Upon closer inspection, the IGN team that ran this test found that one group of visitors in particular - returning visitors - did not react positively to this change. It’s possible that habitual users were no longer able to find the link to videos after it had been removed from its initial position. The lesson: Make sure that you think consider your different audiences when you optimize every channel - if different visitors would benefit from changes or be alienated by others, build that in to your hypothesis for testing.
• #43: So that’s a reminder to optimize your customer acquisition channels - you could uncover a new traffic opportunity through optimization, or make the conversions on an existing channel perform even better.
• #45: A reminder to personalize - it’s easier than you think, and doesn’t require a team of engineers to get started. Start by targeting an experiment towards an audience and measuring the effect.
• #46: This is a well-known example of personalization. Many companies like Amazon
• #47: Personalization is neither a massive engineering effort nor a departure from your current experience. You may not get to the level of personalization of Netflix or Amazon, but you can get to the 80/20 of personalization - that it, 80 percent of the outcome with 20 percent of the investment. Personalization starts with very simple A/B tests. Start by learning which messaging, experiences, flows, etc. work for your audiences. You can begin to do that by targeting A/B tests at specific audiences like return traffic, mobile visitors, customers and non-customers. This is why customer personas are already part of the work towards personalization.
• #48: Eric is a Digital Marketing Manager at Iron Mountain. They’re a records management company - they help companies save, manage, and also securely dispose of sensitive documents. They’re catering to lots of different businesses that have sensitive data: Healthcare companies, legal professionals, small businesses. They implemented a product, Demandbase, to start getting more information about their visitors. It identified customers by IP address and allows them to serve up custom content based on their industry. This personalization was interesting, but Eric felt they were blindly customizing experiences without knowing what was working for their visitors. To measure the impact of customized content for these verticals, he used Optimizely.
• #49: This test used Demandbase to filter IP addresses for specific industry verticals that Iron Mountain markets towards. This would allow Eric to target industry-specific content to first-time visitors, and ideally increase their engagement. This test targeting healthcare industry content (HIPAA regulations, medical records compliance) to new visitors to the homepage. The experiment garnered 123% more engagement with the industry-specific content and offerings. Eric also built out homepage variations for other industry verticals: B2B, SMB, Federal, and Legal with this solution, and has seem similar lifts in those experiments.
• #50: A reminder to personalize - it’s easier than you think, and doesn’t require a team of engineers to get started. Start by targeting an experiment towards an audience and measuring the effect.
• #56: In an optimized marketing process, we make changes, test, measure, and start again. Think about applying this to your customer acquisition channels so that you can continuously optimize and capture compounding gains for your business.
• #57: Mike Wolf leads product design at Trunk Club. They’re an earlier stage company, so he wears a number of hats and does web development, product design, and some one-off projects with the marketing team. All new users of Trunk Club must create an account as well as a style evaluation. This helps their stylist personalize and start sending “trunk” packages to suit their personal tastes.sign up via a simple form, and were met with a long style evaluation - Mike compared it to signing up for car insurance. Previously, the assumptions were that fewer steps were better, since there were fewer dropoff points. Mike wanted to revamp the onboarding process because he was frustrated with the options available for iterating on the current design. He wanted to try something drastically different.
• #58: He created a totally new signup flow that broke the style evaluation into 6 steps. He used Optimizely to redirect half of Trunk Club’s traffic to the new onboarding flow. Essentially, he was taking a huge risk by introducing such a long signup process into their acquisition funnel. Fortunately, Trunk Club has a strong experimental culture, so the team was supportive of the test, and eager to see the results of the test. The results blew away the team’s expectations by more than doubling the rate at which new visitors signed up for Trunk Club (a 133% increase). He continued to optimize the flow to eventually reach a 150% improvement over the previous flow. http://prezi.com/dqbe89kehtjr/my-first-year-at-trunk-club/
• #59: Mike’s experimentation gave him a concrete example of the business value he created for his company during his first year as an employee. Here’s a slide describing his experiment that he showed to his team; he describes his process and the results of the experiment, as well as the downstream effects: 20,000 more customers in the next year, creating at minimum 10M in revenue during their first year as customers. *Possible joke* - We wonder if Mike might be getting a raise? http://prezi.com/dqbe89kehtjr/my-first-year-at-trunk-club/
• #60: Take this as a reminder to be fearless when running tests - but also remember to iterate and persist until you reach your big wins. Remember to be idea driven, as well as data-driven.