Appiterate - SAIF Ignition
0 likes620 views
The document discusses A/B testing for mobile apps, highlighting its importance in increasing conversion rates and user engagement despite challenges like complex coding and app store delays. Examples of successful tests include variations in call-to-action buttons and user interface elements, demonstrating the impact of such tests on app revenue. The case study of a price comparison app illustrates how changes in the review button's text and visual elements can significantly boost user engagement and revenue.
Appiterate - SAIF Ignition
- 1. TEXT SAIF Ignition Bangalore 20th Sep 2013 A/B testing for mobile apps
- 2. WHAT IS A/B TESTING? Version ‘A’ vs Version ‘B’ That’s how the name A/B testing came about. How does it work?
- 3. WEB A/B TESTING - Etsy • In-house A/B testing solution • Results are sometimes surprising! INFINITE SCROLL • Infinite scroll is presumed to be a better experience as you see more items faster • Visitors clicked on • fewer results (-11%) • favourited less (-8%) • fewer items purchased (-22%) Source – Design for continuous experimentation, http://www.slideshare.net/danmckinley/design-for-continuous-experimentation Infinite Scroll
- 4. WEB A/B TESTING INFINITE SCROLL • What went wrong? • Changed multiple things at once • They should have broken-down the feature change into two hypothesis testing validations: • More items is better • Faster is better
- 5. WEB A/B TESTING SEARCH DROPDOWN • Broke-down the process into testing multiple short measurable tests • Decrease the number of items in dropdown • Default to ‘all items’ • Search bars for in-shop and in-page search • Rich auto-suggest • .. and many more such tests!
- 6. WEB A/B TESTING SEARCH DROPDOWN • Most people did not notice the major feature changes • Search dropdown was subsequently withdrawn • Iterative ‘Design – Develop – Measure’ Infinite Scroll Design Search Dropdown Design Develop Measure Develop Measure
- 7. MOBILE A/B TESTING – Why is it needed? • Increase conversion, revenue, engagement metrics • User-segmented and targeted app feature launches • Most mobile apps struggle to make money - faster iterations will genuinely help developers in monetizing their apps
- 8. MOBILE A/B TESTING – Challenges • Complex coding involved • Delays in app store updates • slowing the pace of innovation • Simplicity and ease of web A/B testing not prevalent on mobiles • thus devoid of 'iterative improvements‘ • key growth drivers on the web
- 9. WHY A/B TEST MOBILE APPS? • Everyone A/B tests in real-life: • People try two or more pieces of clothing before buying • You have a test drive of two shortlisted cars • The HiPPO (the Highest Paid Person’s Opinion) method of decision making • Decisions based on such opinions, hunches, guesses and conjectures • Use data in decision making • A/B testing on mobile apps is a great way to increase conversions and improving your key mobile metrics.
- 10. WHAT TO A/B TEST IN NATIVE MOBILE APPS? If it is there in your app, it can be A/B tested! Example use cases: • Call to Action buttons (these include buttons like ‘buy now’, ‘add to cart’, ‘sign up’) • Set any value to the text, size, colors, fonts or its position • App features – Test engagement metrics by having different number of free levels in games • User experience and app navigation sequence tests • Entire screen themes/layouts • Different page opens on tap of a button • Different ways of social media credentialbased sign-ins
- 11. CASE STUDY: PRICE COMPARISON APP The app lets users compare prices of multiple cross-category products from online stores. User engagement metric ‘review’ button •Price link – Increases element (arrow) beside the price Variation A review button with longer text No visual element (arrow) beside the price A review button with shorter text A small arrow beside the price DISTRIBUTION: 50% 50% Price link clicks on 18.1% of screen opens Price link clicks on 22.4% of screen opens Hypotheses: •Review button – Control DESCRIPTION: Aim was to increase conversion on the following: VARIATIONS: revenue in the form of referral fees. •A short or a long text on a •Impact of a small visual RESULTS: Longer text button got Shorter text button got clicked clicked 1.4% times of screen 1.1% times of screen opens opens Price link with a visual element resulted in an increase of 23.4% in revenues for the app! The longer text on the review button is better at driving engagement by 23.1%
- 12. values in-code test (normal) more •Integrate SDK •Pre-specify tests •Pre-specify attribute PRE SETUP TIME less COMPARING TEST TYPES (1/3) slow TIME FOR DEPLOYMENT & ITERATION fast
- 13. COMPARING TEST TYPES (2/3) •Integrate SDK •Pre-specify tests •Change attribute values values data driven test in-code test (normal) more •Integrate SDK •Pre-specify tests •Pre-specify attribute PRE SETUP TIME less live slow TIME FOR DEPLOYMENT & ITERATION fast
- 14. COMPARING TEST TYPES (3/3) •Integrate SDK •Pre-specify tests •Change attribute values PRE SETUP TIME less values wysiwyg data driven test in-code test (normal) more •Integrate SDK •Pre-specify tests •Pre-specify attribute •Bypass app store updates •Drag-n-drop visual interface •Fast and easy deployment live slow TIME FOR DEPLOYMENT & ITERATION fast
- 15. NEW FEATURES Live Tracking of User’s Interaction with the App Content Management System Feature Launches and Segmented Rollouts
- 16. www.appiterate.com “If genius is, as Edison said, 1% inspiration and 99% perspiration, then perhaps innovation is 1% invention and 99% iteration.”