ade_optimizely_user_session_presentation_revised
Download as PPTX, PDF1 like158 views
The document discusses how the author's company initially thought their product tour was successful because it increased engagement metrics. However, they later learned engagement may not be the best metric. Bounce rates increased and some users said the tour was annoying. Further analysis found the tour actually decreased non-tour content clicks by 20% while increasing them by 36% without the tour. The author advocates using multiple metrics like clicks, pageviews, and subjective user feedback to fully understand optimizations rather than relying solely on engagement metrics.
ade_optimizely_user_session_presentation_revised
- 1. How I learned to hate ‘engagement’ (and love subjective feedback)
- 2. Title text Arthur Etchells Director of Product Management Imaging Resource
- 3. Our product looked like this
- 5. We added a cool product tour shortly after launch
- 8. ‘Engagement’ was through the roof! People loved it!
- 9. “Lorem ipsum dolor sit amet, consectetur adipiscing elit, set eiusmod tempor incidunt et labore et dolore magna aliquam.” – First Name Last Name, Attribution Or did they?
- 10. “Lorem ipsum dolor sit amet, consectetur adipiscing elit, set eiusmod tempor incidunt et labore et dolore magna aliquam.” – First Name Last Name, Attribution Or did they? Was ‘engagement’ what we wanted?
- 11. • Yes, engagement was great (Optimizely)
- 12. • Yes, engagement was great (Optimizely) • Bounce rates were not (Google Analytics)
- 13. • Yes, engagement was great (Optimizely) • Bounce rates were not (Google Analytics) • Months pass
- 14. • Yes, engagement was great (Optimizely) • Bounce rates were not (Google Analytics) • Months pass • Some internal feedback (that tour is annoying!) …
- 15. • Yes, engagement was great (Optimizely) • Bounce rates were not (Google Analytics) • Months pass • Some internal feedback (that tour is annoying!) … • Led to further research…
- 16. Title Text • Crazy Egg • In-page click analysis
- 19. • Bullets only • Up to five items • One to four words • First letter is capitalized • Periods are not used
- 20. • Bullets only • Up to five items • One to four words • First letter is capitalized • Periods are not used
- 21. • Bullets only • Up to five items • One to four words • First letter is capitalized • Periods are not used
- 22. The Question
- 23. Do users interact more with our content* after viewing a product tour?
- 24. *non-product tour content Do users interact more with our content* after viewing a product tour?
- 25. The Experiment
- 26. Two Variations
- 29. Two Metrics
- 30. Engagement*
- 31. Engagement* *But not as the primary goal
- 32. Clicks on non-product tour content
- 33. Results
- 35. Title Text • Engagement • Dropped by 20.4%
- 36. Title Text • Clicks on non-product tour links
- 37. Title Text • Clicks on non-product tour links • Increased by 36.3%
- 38. Lessons Learned
- 39. 1
- 40. Be careful what you wish for
- 41. Be careful what you wish optimize for
- 42. Engagement may not be the best metric to optimize
- 47. • Yes, engagement was great (Optimizely) • Bounce rates were not (Google Analytics)
- 48. • Yes, engagement was great (Optimizely) • Bounce rates were not (Google Analytics) Any click
- 49. • Yes, engagement was great (Optimizely) • Bounce rates were not (Google Analytics) Pageviews and events Any click
- 52. Optimizely: Not a fan of the engagement metric either
- 53. Engagement may not be the best metric to optimize
- 54. Engagement may not be is not the best metric to optimize
- 55. 2
- 56. Make sure product tours are actually doing what you hope they are
- 57. Make sure product tours are actually doing what you hope they areThere should be a high bar for anything you put between users and your content
- 58. 3
- 59. Take a holistic approach to optimization efforts
- 60. Take a holistic approach to optimization efforts Subjective feedback can let you uncover issues earlier
- 61. User feedback sessions In-page feedback prompt HeatmapsAnalytics Biz goals: revenue, lead gen Objective Subjective
- 62. User feedback sessions In-page feedback prompt HeatmapsAnalytics Biz goals: revenue, lead gen Objective Subjective
- 63. User feedback sessions In-page feedback prompt HeatmapsAnalytics Biz goals: revenue, lead gen Objective Subjective
- 64. • Those were the tools we used, but there are lots of options…
- 65. • Those were the tools we used, but there are lots of options… • Analytics (Omniture/Google Analytics/Mixpanel/Heap)
- 66. • Those were the tools we used, but there are lots of options… • Analytics (Omniture/Google Analytics/Mixpanel/Heap) • Heatmaps (ClickTale/CrazyEgg/HotJar)
- 67. • Those were the tools we used, but there are lots of options… • Analytics (Omniture/Google Analytics/Mixpanel/Heap) • Heatmaps (ClickTale/CrazyEgg/HotJar) • In-page feedback forms (Qualaroo/HotJar)
- 68. • Those were the tools we used, but there are lots of options… • Analytics (Omniture/Google Analytics/Mixpanel/Heap) • Heatmaps (ClickTale/CrazyEgg/HotJar) • In-page feedback forms (Qualaroo/HotJar) • User feedback sessions (In-person/UserTesting)
- 69. Subjective feedback can allow you to uncover usability problems earlier
- 70. Subjective feedback can allow you to uncover usability problems earlierWhat does that look like in practice?
- 73. “Lorem ipsum dolor sit amet, consectetur adipiscing elit, set eiusmod tempor incidunt et labore et dolore magna aliquam.” – First Name Last Name, Attribution Questions?