Conversion Optimization: The World Beyond Headlines & Button Color
2 likes801 views
The document discusses several testing strategies for optimizing conversion rates, including testing quotas, payment methods, offline processes, onboarding experiences, and core product interactions. Key insights emphasize the importance of user experience and well-structured a/b testing, revealing that understanding user interactions and maintaining communication between teams can significantly impact sales. The author provides crucial guidelines for successful testing, underlining the need to adapt and iterate based on data-driven findings.
![@user.card_limit
=
ab_test("user_card_limit_May_2012",
[8,
15],
:conversion
=>
"purchase")](https://image.slidesharecdn.com/opticon2014conversionoptimizationtheworldbeyondheadlinesandbuttoncolor-140501133953-phpapp01/85/Conversion-Optimization-The-World-Beyond-Headlines-Button-Color-5-320.jpg)
![Single
Most
Important
Lesson:
One
Line
of
Code
@user.card_limit
=
ab_test("user_card_limit_May_2012",
[8,
15],
:conversion
=>
"purchase")](https://image.slidesharecdn.com/opticon2014conversionoptimizationtheworldbeyondheadlinesandbuttoncolor-140501133953-phpapp01/85/Conversion-Optimization-The-World-Beyond-Headlines-Button-Color-41-320.jpg)
Conversion Optimization: The World Beyond Headlines & Button Color
- 1. Conversion Op,miza,on: The World Beyond Headlines and Bu:on Color Patrick McKenzie Founder Kalzumeus So>ware @pa,o11
- 2. Low hanging fruit tastes great, but the rest is worth the work, too.
- 3. Test Idea #1 Quotas / Tier Levels / etc
- 5. @user.card_limit = ab_test("user_card_limit_May_2012", [8, 15], :conversion => "purchase")
- 6. What Did We Learn From That Test? • ~60% increase in sales going from 15 -‐> 8 cards • Synergis,c effects can occasionally actually happen. – More on that later. • You don’t know what you don’t know un,l you test for it. This post on HN was me losing $100,000:
- 7. Key Points With TesPng Quotas • Grandfather in exis,ng users. • Set quota levels intelligently. – Quotas should segment people by willingness to pay – Break out a histogram of exis,ng accounts • Be ready to tolerate small up,ck in support load. • Your product is s%ll too cheap.
- 8. Test Idea #2 Payment Methods / Checkout Flow
- 9. Threw This Together In An Hour (Oops)
- 11. What We Learned From This Test • A single A/B test can help your boZom line more than a product launch. • Dealing with synergis,c effects in tes,ng can occasionally be maddening. – First a:empts at implemen,ng on-‐site checkout had null result a>er null result – Combina,on of on-‐site checkout plus redesign plus aforemen,oned card limit test plus hillclimbing won big • The data is never wrong, but some,mes the data isn’t telling you what you think it is.
- 12. Key Points With TesPng Checkout Flow • Make absolutely sure you don’t break this! • Have UX-‐focused team (product / marke,ng / etc) go over every step of checkout funnel with zero tolerance for bad UX. • Checkout flows go “stale” at many successful companies. Schedule a deep dive on them twice a year, at least. • Upsells and cross selling are classics for a reason!
- 13. Test Idea #3 Offline Business Processes
- 15. TesPng degree of sales which maximizes conversion of leads
- 16. What We Learned From This Test • Important to communicate with sales/CS. – Allay fears about being replaced, leads creamed, etc. – Demonstrate it directly makes them money. • Lead to resource-‐intensive but hugely successful lead-‐scoring efforts. • Jus,fied very expensive interven,ons like e.g. onboarding help for everybody.
- 17. Key Points With TesPng Offline Processes • Your so>ware/workflow has to close the loop back to the tes,ng so>ware. This can be tricky. • Virtually no CRM or A/B tes,ng so>ware accurately models enterprise decisionmaking process. You might be forced to op,mize for proxy conversions. • Volumes required for sta,s,cal significance are really punishing, so some,mes we “proxy and pray.”
- 18. Test Idea #4 Product On-‐boarding / FRE
- 19. What Engineers Will Ship If You Let Them
- 20. Key Goals of Onboarding Process • Get user to a concrete success as soon as possible. • For more complex sales, lay groundwork for future success by e.g. invi,ng in team or gepng permission to follow-‐up. • For B2C, especially in social, you live and die on virality during the onboarding experience.
- 23. What We Learned From This Test • Users who start onboarding tour 25% to 100%+ more likely to purchase so>ware • Users some,mes have visceral reac,on to elements of the onboarding experience. – The wise-‐alecky Kiwi – “Beau,ful So>ware Lady Susan” • Engineers prefer in-‐app onboarding but it is expensive as heck to build and does not always jus,fy itself versus e.g. a Youtube video.
- 24. Key Points With TesPng Offline Processes • Your so>ware/workflow has to close the loop back to the tes,ng so>ware. This can be tricky. • Virtually no CRM or A/B tes,ng so>ware accurately models enterprise decisionmaking process. You might be forced to op,mize for proxy conversions. • Volumes required for sta,s,cal significance are really punishing, so some,mes we “proxy and pray.”
- 25. Test Idea #5 Core Product InteracPons
- 26. Too Complicated
- 27. 80% of Users Use 20% of Features
- 30. What We Learned From This Test • Just hiding the advanced features increased task success by 10%, which directly improved sales • Further jump of 15% when we did the redesign – Absolutely no increase in sales… – … but we saw in the funnel that this was solely due to less people hipng the trial limits … – … which we were able to profitably exploit with the quota limit test described earlier
- 31. Key Points With TesPng Core InteracPons • Test the en,re funnel. Some,mes task success doesn’t (unfortunately) help bo:om line metrics. • A good understanding of where people spend their ,me/energy is vital – test the 90% interac,on, not the 2%.
- 32. ROI on ~$500 A Month TesPng Budget 0.00% 0.50% 1.00% 1.50% 2.00% 2.50% 3.00% 3.50% February, 2010 March, 2010 April, 2010 May, 2010 June, 2010 July, 2010 August, 2010 September, 2010 October, 2010 November, 2010 December, 2010 January, 2011 February, 2011 March, 2011 April, 2011 May, 2011 June, 2011 July, 2011 August, 2011 September, 2011 October, 2011 November, 2011 December, 2011 January, 2012 February, 2012 March, 2012 April, 2012 May, 2012 June, 2012 July, 2012 August, 2012 September, 2012 October, 2012 November, 2012 December, 2012 January, 2013 February, 2013 March, 2013 April, 2013 May, 2013 June, 2013 July, 2013 August, 2013 September, 2013 October, 2013 November, 2013 December, 2013 January, 2014 February, 2014
- 33. Let’s Talk TesPng Challenges
- 34. A/B TesPng Across System Boundaries
- 35. Not A Hard Problem Our Marke,ng Site A/B
- 36. Gets A Wee Bit Trickier With 2 Sites Our Marke,ng Site A/B The Web Applica,on
- 37. Total Freaking Madness Our Marke,ng Site A/B The Web Applica,on iPhone app Offline processes Android app Partner site Content minisite Offsite checkout Japanese subsidiary
- 38. Sanity Restored Consumer Applica,ons Iden,ty & A/B Manager
- 39. Lessons Learned With This • Think about strategy for transi,oning people between systems – easier to build into design earlier than later • Cookies with unique iden,,es are a stopgap, not a solu,on. You need to understand the sales process. • This is a good opportunity to standardize on an A/B tes,ng tool and workflow… or you’ll hate your life • This is a “high class problem” which can be deferred un,l $50M+ in revenue, but when you solve it, it is going to cost serious ,me/money.
- 40. Increasing A/B TesPng AdopPon In Engineering and MarkePng Teams
- 41. Single Most Important Lesson: One Line of Code @user.card_limit = ab_test("user_card_limit_May_2012", [8, 15], :conversion => "purchase")
- 42. Other Lessons Learned • Organiza,on should learn rather than guess be:er. – Best investment ever: organiza,on-‐wide wiki. – Next best: weekly “What we tried this week” email. • Engineers love A/B tes,ng when you describe it like a puzzle or a video game, with bragging rights. • Marketers love A/B tes,ng when it’s posi,oned as a way to increase their status in the organiza,on rather than a replacement for their talents.
- 43. Things To Never Be Without • CS should be able to force-‐override a user’s par,cipa,on in an A/B test (and, relatedly, “see what users is seeing” is great if you can get it). • A/B tests deeper than surface level need unit/etc tests. Make sure your framework supports it. • Test switcher UI in development/staging is a huge win. • URL query parameter to flip tests a huge win.
- 44. Thanks for Listening! • I love talking about this and you cannot possibly waste my ,me: @pa,o11 or patrick@kalzumeus.com • Free email mini-‐course on CRO in so>ware companies: h:p://bit.ly/patrick-‐op,mizely • Liked this presenta,on? I deliver something like this twice a month, free, to my email list. h:p://training.kalzumeus.com to sign up
- 45. Conversion Op,miza,on: The World Beyond Headlines and Bu:on Color Patrick McKenzie Founder Kalzumeus So>ware @pa,o11