Behavior Based Approach to Experiment Design
- 1. Head to Irrationallabs.org for more information on applying behavioral economics THIS IS A GUIDE TO DESIGNING EXPERIMENTS
- 2. Now with loss aversion in mind let’s proceed to the next step... WARNING Be sure to fill in all the blanks before proceeding to the next step. Skipping this step will prevent you from getting the most out of this guide.
- 4. Identify Key Behavior1 What is the key behavior that you want your users to do? (eg. adopt negative keywords)
- 5. (what you will measure to see the effect of your test) Identify Key Behavior1 EXPERT TIP Ask around, does everyone involved with your project agree that this is the most important behavior to measure? Key Behavior = Measured Variable
- 7. any factor that decreases willingness to act on the key behavior There are 2 types of behavioral barriers: Identify Behavioral Barriers2 Behavioral Barriers factors that make it difficult for a user to engage in the key behavior HEADWINDS factors that make the key behavior not sufficiently motivating TAILWINDS
- 8. What are the barriers preventing people from doing the key behavior that you want them to do? (list as many as you can think of, both tailwinds and headwinds) Identify Behavioral Barriers2
- 9. Identify Behavioral Barriers2 friction takes too much time, complex sign up no perceived benefit don’t believe the benefit, there is no benefit, benefit is too far in the future HEADWIND TAILWIND
- 10. Identify Behavioral Barriers2 EXPERT TIP When thinking about tailwinds, think about enhancing current benefits and creating new ones If you don’t know behavioral barriers and benefits, check out our workbooks
- 11. CHOOSE A BARRIER
- 12. Choose A Barrier3 Choose the behavioral barrier that you think detracts most from people not exhibiting the key behavior, and explain it in the following format...
- 13. Choose A Barrier3 The reason people are not currently partaking in is because they KEY BEHAVIOR BEHAVIORAL BARRIER
- 14. Choose A Barrier3 The reason people are not currently partaking in adopting negative keywords is because they put it off until later KEY BEHAVIOR BEHAVIORAL BARRIER
- 16. Identify BE Principles4 EXPERT TIP If you don’t know BE principles, check out our workbooks
- 17. Identify BE Principles4 Using BE principles, what are some ways that you can remove this barrier? (list as many as you can think of)
- 18. Identify BE Principles4 pre-commitment put it on their calendar in-your-way instructions have a pop-up at login urgency provide a deadline B.E. PRINCIPLE B.E. PRINCIPLE B.E. PRINCIPLE
- 20. Choose A Principle5 Choose the behavioral economic principle that you think is most likely to remove the barrier
- 21. Choose A Principle5 By using we can increase the number of people who B.E. PRINCIPLE KEY BEHAVIOR
- 22. Choose A Principle5 By using urgency we can increase the number of people who adopt negative keywords B.E. PRINCIPLE KEY BEHAVIOR
- 24. Write A Hypothesis6 If we do this… then this will happen… because...
- 25. Write A Hypothesis6 If we add a sense of urgency then people will be more likely to take action to add negative keywords, because urgency gives people a clear timeline in which they must complete the action in order to get the incentive.
- 27. Create Test Conditions7 Design your test below. Start with two conditions: What will each group of participants see? Write out the exact text. Will you use images? If so, add the exact images. CONTROL TEST
- 28. Make your ad more relevant. Adopt keywords and receive $50 in AdWords Credit Make your ad more relevant. Adopt keywords in the next 7 days and receive $50 in AdWords Credit Create Test Conditions7 CONTROL CONDITION TEST CONDITION Test Condition = Control Condition + One Key Variable you are testing *
- 29. Create Test Conditions7 EXPERT TIP Restrict to only one variable in the test condition. Imagine you include two variables in the test condition (e.g., urgency and amount of incentive) and the test condition works twice as well as the control condition. Why did it happen? YOU DON’T KNOW (it could have been urgency, OR the amount of the incentive, OR a combination of both)
- 30. Create Test Conditions7 GURU TIPS *use at your own risk! You can have more than two conditions: The status quo is never the control. You can have more than one variable between test conditions if they are testing the same principle. If this is the last experiment you are ever running, then you have our permission to test different concepts and multiple key variables. CONTROL TEST CONDITION 1 STATUS QUO TEST CONDITION 2
- 31. Do I have a control condition? ✓ Do I have only one key variable between conditions? Do participants understand the key behavior that I want them to do? Do participants understand how to do the key behavior? Are my instructions clear? Quick Review
- 33. Select Participants8 Let’s do a quick Q&A...
- 34. Select Participants8 Who will take part in your experiment? All of your customers? Some of your customers? Make sure participants are representative of your entire customer base. If you only test your most/least loyal customers, the results may not generalize to other customers.
- 35. How will you decide which participant is in which condition? Participants should be randomly assigned. There should not be a special reason why someone is in your control or test condition. Select Participants8
- 36. = Test Condition (6) Select Participants8 EXPERT TIP Use spreadsheets to randomize your participants. Create as many random numbers as you have participants. Then assign each participant a number and list the participants in ascending or descending order. 12 Participants = Control Condition (6)
- 37. Select Participants8 How many participants will take part in your experiment? Think about the number of people who are going to do the key behavior. Think about what relevant funnels might shrink your sample size. You should target 250+ per condition. IMPRESSIONS > CLICKS > =
- 38. Select Participants8 How many participants in each condition? Each condition should have the same number of participants.
- 39. REFLECT
- 40. Reflect9 Take some time to reflect on your experiment (before & after) What do you expect will happen? Write down your predictions and try to plot the data. Make sure you know how to analyze the data. Do you believe this experiment will prove your hypothesis? Plot your data. Compare results to your prediction. Did the experiment support your hypothesis? Do you need further experimentation? BEFORE AFTER
- 41. 2 Final Review Identify Key Behavior 1 3 4 5 6 7 8 Identify Behavioral Barriers Choose a Barrier Identify BE Principles Choose a Principle Write a Hypothesis Create Test Conditions Select Participants 9 Reflect
- 42. POP QUIZ!
- 43. What’s wrong with this experiment?Q1 Start being active today! Sign up for the Hard Body Gym today! Start being active today! Sign up for the Hard Body Gym today and get 20% off your membership I want to increase fitness activity among my employees. I predict that giving people 20% off their gym membership will get them to be more active. I create the conditions below. Half of my employees are randomly assigned to each condition. I measure how many of them sign up for the gym membership. TESTCONTROL
- 44. What’s wrong with this experiment?Q1 Start being active today! Sign up for the Hard Body Gym today! CONTROL Start being active today! Sign up for the Hard Body Gym today and get 20% off your membership TEST Measuring the wrong key behavior Gym membership ≠ increased fitness activity
- 45. Donate today and save a life CONTROL Q2 I’m curious to see which plea will get people to donate more money. I create the two conditions below. I randomize my 4,000 participants, and send 2,000 one version and the other 2,000 the other version. I measure the number of people who donate, and how much they donate. What’s wrong with this experiment? Donate today and get a tax write off TEST
- 46. Donate today and save a life TESTCONTROL Q2 What’s wrong with this experiment? Donate today and get a tax write off More than one key variable “Save a life” + Picture
- 47. What’s wrong with this experiment?Q3 Base Package: $200 Premium Package: $350 CONTROL Base Package: $200 Premium Package: $350 Premium Plus Package: $400 I run an enterprise software company, and want to see which type of pricing will get more companies to sign up for my product. I predict that having 3 options instead of just 2 will increase my revenue. I send one version to my customers in Burlington, VT and the other to my customers in Los Angeles, CA. TEST
- 48. What’s wrong with this experiment?Q3 Conditions ≠ randomly assigned Customers in different areas may have different preferences Base Package: $200 Premium Package: $350 CONTROL Base Package: $200 Premium Package: $350 Premium Plus Package: $400 TEST
- 49. TESTCONTROL What’s wrong with this experiment?Q4 I run a financial management company and want to see if people sign up more when they know how much time it will actually take them to sign up. Here are my conditions: Without rebalancing quarterly, you may be missing out on potential money Get more money from rebalancing in just 15 minutes each quarter
- 50. TESTCONTROL What’s wrong with this experiment?Q4 Without rebalancing quarterly, you may be missing out on potential money Get more money from rebalancing in just 15 minutes each quarter More than one key variable “Get more money” + “15 minutes”
- 51. Congratulations! You have passed designing experiments 101 To become an expert contact:
- 52. GURU QUESTIONS
- 53. We notice Airbnb guests really value a quiet environment. How do we get hosts to decrease noisiness?
- 54. Let’s follow the steps 2 Identify Key Behavior 1 3 4 5 6 7 8 Identify Behavioral Barriers Choose a Barrier Identify BE Principles Choose a Principle Write a Hypothesis Create Test Conditions Select Participants 9 Reflect
- 55. Identify Key Behavior1 What is the key behavior that you want your users to do? KEY BEHAVIOR
- 56. Identify Key Behavior1 What is the key behavior that you want your users to do? Decrease noisiness KEY BEHAVIOR
- 57. Identify Behavioral Barriers2 What are the barriers preventing people from doing the key behavior that you want them to do? (list as many as you can think of, both tailwinds and headwinds)
- 58. Identify Behavioral Barriers2 don’t want to be quiet are not actively thinking about it no incentive to be quiet BEHAVIORAL BARRIERS
- 59. Choose A Barrier3 KEY BEHAVIOR BEHAVIORAL BARRIER The reason people are not currently partaking in is because they (behavioral barrier)
- 60. Choose A Barrier3 KEY BEHAVIOR BEHAVIORAL BARRIER The reason people are not currently partaking in decreased noisiness is because they have no incentive to be quiet (behavioral barrier)
- 61. Identify BE Principles4 Using BE principles, what are some ways that you can remove this barrier? (list as many as you can think of)
- 62. Identify BE Principles4 concreteness email reminders explaining the importance of providing a quiet environment incentives providing star rating on noisiness B.E. PRINCIPLE B.E. PRINCIPLE
- 63. Choose A Principle5 Choose the behavioral economic principle that you think is most likely to remove the barrier
- 64. Choose A Principle5 By using incentives we can increase the number of hosts who decrease noisiness KEY BEHAVIOR B.E. PRINCIPLES
- 65. Choose A Principle5 By using incentives we can increase the number of hosts who decrease noisiness KEY BEHAVIOR B.E. PRINCIPLES
- 66. Write A Hypothesis6 If we do this… then this will happen… because...
- 67. Write A Hypothesis6 If we add incentives, then hosts will be more likely to take action to decrease noisiness, because incentives nudge people to complete an action in order to get the incentive.
- 68. Create Test Conditions7 CONTROL TEST CONDITION 1 TEST CONDITION 2 Design your test below
- 69. Create Test Conditions7 Noise level Control - rate noise level
- 70. Create Test Conditions7 Noise level Condition 1 - rate noise level + gift incentive (mug)
- 71. Noise level Create Test Conditions7 Condition 2 - rate noise level + financial incentive ($10)
- 72. Create Test Conditions7 What is wrong with these conditions?
- 73. Select Participants8 Select 1,200 randomly generated hosts in the US 400 hosts per condition
- 74. Reflect9 Take some time to reflect on your experiment (before & after) What do you expect will happen? Write down your predictions and try to plot the data. Make sure you know how to analyze the data. Do you believe this experiment will prove your hypothesis? Plot your data. Compare results to your prediction. Did the experiment support your hypothesis? Do you need further experimentation? BEFORE AFTER
- 75. Head to irrationallabs.org to download our behavioral economics workbooks