Drippler's A/B test library
Download as PPTX, PDF2 likes838 views
The document presents an overview of A/B testing in Android, discussing its importance for data-driven decision-making and user engagement. It details the process of setting up tests, including hypothesis formulation and analyzing results through the use of the Amazon A/B Testing SDK, along with code examples. Additionally, it emphasizes the significance of confidence levels in results assessment and offers insights into executing multiple experiments effectively.
Drippler's A/B test library
- 1. A/B Testing in Android Nir Hartmann Drippler Droidcon Tel-Aviv 2014
- 2. Why do we need A/B Testing? • Tests takes the guesswork out • Enables data-backed decisions • Enhances engagement and retention
- 3. Road map • What is an A/B test ? • Segmentation • Multiple Experiments
- 4. What is an A/B Test ? • Case study – onboarding screen
- 5. Define the test • Hypothesis – The layout with the Google+ button at the left will increase the number of total registered users. • Goal – A registered user (the user can skip registration). • View event - Login fragment onCreate(). - Login activity onCreate(). • Variables – Facebook button position (left or right). • Participants – New users.
- 6. Amazon A/B Testing SDK • Very customizable, you can do just about anything as long as you know what it is you want to do • It’s free • Drippler created an open source library that simplify the process
- 7. Setup the A/B test • Setup identifier – https://developer.amazon.com/al/index.html
- 8. Setup the A/B test • Create a project
- 9. Setup the A/B test • Create the test
- 10. Dive into the code • https://github.com/Drippler/ABTester public class MyApplication extends Application { @Override public void onCreate() { super.onCreate(); ABTester.init(getApplicationContext(), "my_public_key", "my_private_key"); } }
- 11. private void initLoginActivityTest() { try { ABTester.syncPreFetch( TimeUnit.SECONDS.toMillis(15), new ABTest("Login page test", false, "Facebook is first”) ); } catch (TimeoutException e) { // Couldn't reach amazon servers } } Fetch the test
- 12. Login Fragment @Override public void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); setRetainInstance(true); ABTester.recordEvent( "Login fragment shown", false); }
- 13. Login Fragment @Override public View onCreateView(LayoutInflater inflater, ViewGroup container, Bundle savedInstanceState) { boolean shouldShowFacebookFirst = ABTester.getBoolean("Login page test", "Facebook is first", false); if (shouldShowFacebookFirst) return inflater.inflate(R.layout.facebook_first, null); else return inflater.inflate(R.layout.google_first, null); }
- 14. Report goal event public void onUserLoggedIn() { ABTester.recordEvent("Sign in", false); } @Override protected void onPause() { super.onPause(); /* Submit the events that were previously stored locally. * asynchronously * call it in the onPause() method of an activity */ ABTester.submitEvents(); }
- 15. Analyze the results Variation Views Conversions Conversion rate Change Google+ first 1064 320 30.08% Facebook first 1043 250 23.97% -20.30%
- 16. Dice experiment Goal: maximize the amount of 3’s we get in a 200 dice roll
- 17. Dice experiment Hypothesis: wearing a hat will increase the chance to roll a 3
- 18. Analyze the results Variation Views Conversions Conversion rate Change Hat off 200 31 15.50% Hat on 200 38 19.00% +22.58% Conversion is never a single number.
- 19. Confidence level • Measure the reliability of an estimate – The confidence levels help us understand if the results are different merely by chance or by reason • 95% confidence level is considered good
- 20. Analyze the results Variation Views Conversions Conversion rate Change Google First 1064 320 30.08% ± 2.32% Facebook First 1043 250 23.97% ± 2.18% -20.30% • Confidence level of 99%
- 22. Choose the best variation • Launch – Choose the winning variation – Control the percentage of customers that receive a new feature
- 23. Road map • What is an A/B test ? • Segmentation • Multiple Experiments http://visualwebsiteoptimizer.com/split-testing-blog/wp-content/uploads/2010/10/2010.09.10.ab_.png
- 24. Segmentation
- 25. Define the test • Hypothesis – Coloring the “Rate” button, will increase the button’s click rate • Goal – Click event on the “Rate” button • View event – RateUsDialogFragment show(); • Variables – “Rate” button color • Participants – All users
- 26. Create the test
- 27. Rate us DialogFragment public class RateUsDialog extends DialogFragment { public static void show(FragmentManager fm, int color) { RateUsDialog rateUs = new RateUsDialog(); Bundle extras = new Bundle(); extras.putInt(“color”, color); rateUs.setArguments(extras); rateUs.show(fm, “my tag”); ABTester.recordEvent("Rate us dialog shown", false); }
- 28. Rate us DialogFragment @Override public Dialog onCreateDialog(Bundle savedInstanceState) { int color = getArguments().getInt("color"); return createColoredDialog(color); }
- 29. Rate us DialogFragment private Dialog createColoredDialog(int color) { ... .setPositiveButton("Rate", new OnClickListener() { @Override public void onClick(DialogInterface dialog, int which) { ABTester.recordEvent("Rate button click", false); } }); return myDialog;
- 30. Rate us test 1) Asynchronously prefetching SplashActivity Default timeout is 60 seconds, and can be overriden by using preFetch(long timeout, ABTest... Test) ABTester.preFetch( new ABTest("Rate us test", false, "Rate button color") );
- 31. Rate us test 2) Show the dialog String fetchedColor = ABTester.getString( "Rate us test", "Rate button color", "#F5F5F5"); int color = Color.parseColor(fetchedColor); RateUsDialog.show(getFragmentManager(), color); 3) Submitting the results onPause() ABTester.submitEvents();
- 32. Analyze the results Variation Views Conversions Conversion rate Change Confidence Control (white) 865 234 27.05% ± 1.51% Variation A (green) 904 250 27.65% ± 1.49% -0.2% Variation B (red) 830 230 27.71% ± 1.55% +2.4% 51% What can I do with these results?
- 33. Segmentation Variation Views Conversions Conversion rate Change Confidence Control (white) 432 92 21.30% ± 1.97% Variation A (green) 464 165 35.56% ± 2.22% +66.9% 98.7% Variation B (red) 420 120 28.57% ± 2.20% +34.1% Variation Views Conversions Conversion rate Change Confidence Control (white) 433 142 32.79% ± 2.26% +22.2% 97.1% Variation A (green) 440 85 19.32% ± 1.88% -27.9% Variation B (red) 410 110 26.83% ± 2.19% -18.8% Under 40 Over 40
- 34. Define the test • Hypothesis – Coloring the “Rate” button, will increase the button click rate • Goal – Click event on the “Rate” button • View event – RateUsDialogFragment show(); • Variables – “Rate” button color • Participants – All users
- 35. Define the test • Hypothesis – Coloring the “Rate” button, will increase the button click rate • Goal – Click event on the “Rate” button • View event – RateUsDialogFragment show(); • Variables – “Rate” button color • Participants – Age specific tests
- 36. Setup the A/B test • Create a segment
- 38. Assign the segment • In your code before the fetch – ABTester.addDimension(”age", myAge); • Dimension will be remembered forever • ABTester library will automatically add a “percentile” dimension – ABTester.addDimension(“percentile”, new Random().nextInt(100));
- 39. Road map • What is an A/B test ? • Segmentation • Multiple Experiments http://unbounce.com/a-b-testing/shocking-results/
- 42. Multiple Experiments • Serial tests – Run the tests one after the other, without the need to redistribute your app – More accurate but takes more time ABTester.preFetch( new ABTest("Rate us test", false, "Rate button color", "Rate actionbar icon") );
- 43. Multiple Experiments • Parallel tests – Run the tests together, increasing the ‘noise’ for dependent tests – Faster ABTester.preFetch( new ABTest("Rate us button", false, "Rate button color”) new ABTest("Rate us actionbar", false, "Rate button icon”) );
- 44. Not a replacement for common sense
- 45. Thanks, any questions ? Don’t overthink it, use Drippler’s A/B test library https://github.com/Drippler/ABTester Nir Hartmann, nhartmann@drippler.com Drippler Droidcon Tel-Aviv 2014