Mobile research best practices

Mobile Research & Survey Design
Best Practices Guide
www.apptentive.com @apptentive

WhyDoMobileResearch?
Mobile research is a powerful and cost-
effective tool for taking the guesswork
out of your mobile strategy.
For a mobile-first company, customer
satisfaction is the single most important
driver of revenue. But before you can
hope to delight your mobile customers,
you need a mechanism for dissecting their experience with your app and give them a
voice to express suggestions and critiques.
Consider the following questions that mobile research can address:
➢ How do customers like your app?
➢ Are there any pain points in the customer journey?
➢ Is there an existing demand for the feature you want to introduce in your next
update?

WhyDoMobileResearch?
Mobile research allows you to:
➢ Open up communication with your mobile customers
➢ Identify customer segmentation trends and pain points
➢ Improve mobile retention and customer loyalty by asking customers what they’d
like to see in your app, and acting on those insights
➢ Circumvent negative app store ratings by addressing customer issues before they
make their way to a public channel
➢ Extend your sample size – if done right, mobile surveys have dramatically greater
response rates than web, phone, or email surveys

GettingStarted
So how do you begin to design a survey and integrate it into your app?
Start by asking yourself these three questions that we will go over in this
presentation:
➢ What are you looking to learn? What are your research
questions?
➢ What sort of data provides this insight and what should
you be asking to get this data?
➢ Who is your audience and how will you reach them?

WhatAreYouLookingToLearn?
Before designing your mobile research instrument, take a step back and consider
your objective for doing this research. Is your research purely exploratory or do you
have a specific research question you’re looking to address?
Your sampling criteria and survey design should align with your objective – asking
the right questions and getting them in front of those in the best position to answer
them.
Research questions can be attitudinal, behavioral, demographic, or technical.
Attitudinal: How do new users like your app?
Behavioral: How do users interact with your app? What are the most common use
cases?
Demographic: Which age bracket is your app most popular with?
Technical: How can this app be improved?

WhatSortofDataProvidesTheseInsights?
Allow what you want to learn from your research and what data you want to collect
and analyze to dictate the questions you ask.
Some question types are inherently better suited for particular cases.
For example, consider using an open-ended question if you’re looking to open up
communication and welcome new suggestions.
Likewise, use a closed-ended question with a limited number of predefined response
choices if you’re looking for more specific information on a particular attribute of the
app or customer journey.

WhatSortofDataProvidesTheseInsights?
Survey questions can be classified as:
Open-ended (i.e.: Providing a textbox for the respondent to type his or her answer)
➢ Exploratory in nature
➢ Less likely to result in bias from leading questions/response choices
➢ Provides qualitative responses similar to a focus group
➢ Can be time-consuming to answer, particularly when using a mobile device,
leading to lower response rates
Closed-ended (i.e.: Multiple choice questions and rating scales with pre-defined
response choices (see next slide))
➢ Can provide both qualitative and quantitative responses
➢ Questions typically take less time to answer and experience higher response
rates
➢ Questions and response construction requires more care remove bias
Mixed (i.e.: A multiple choice question with a fill-in ‘Other’ option)
➢ Allows room to write in answers that were not considered when the survey was
designed – may uncover new customer needs/sentiments

TypesofClosed-EndedQuestions
Dichotomous Multiple Choice Likert Scale Semantic
Differential
Example Do you like this
app?
-Yes
-No
How many times a week
do you use this app?
- 0-1
- 2-4
- 6-7
- 8+
I would recommend this
app to a friend.
-Strongly agree
-Agree
-Undecided
-Disagree
-Strongly disagree
How would you
describe your
experience with this
app?
Good 1-2-3-4-5-6-7
Bad
Uses Can be used to
direct
respondents to
different follow-
up questions
based on their
information
Gathering generalized
data in a way that takes
very little time away
from the customer
experience
Allows for operationalizing and quantifying
customer attitudes and perceptions around an
attribute of your mobile app
Precautions Respondents may
not see the
question as black
and white and
desire more
choices
Multiple choice options
commonly experience
overlap.
I.e.: How many times a week do you use
this app?
(a) 0-2 (b) 2-4 (c) 4-6 (d) 6+
Both (a) and (b) are
valid choices for a
customer who uses the
app twice a week.
Commonly leads to a central tendency bias,
where respondents are hesitant to choose an
option at either extreme, and instead select a
middle-of-the-road response. As a result, the
deviation in responses may understate the
deviation in actual opinions.

WhoIsYourAudience?
As you begin to think about mobile research, first consider the criteria for the sample
from whom you want to collect the research from and how you will reach them
within the app.
Common mobile research samples include:
Opt-in (no sampling criteria)
Pros:
➢ Survey can be easily integrated as a link in your app’s navigation with no third
party tools or prompts
Cons:
➢ Can lead to selection bias as those who opt-in may not share fundamentally
different views/attitudes than those who do not opt-in
➢ Opt-ins have the lowest response rate and the survey link can be difficult for
potential respondents to find

WhoIsYourAudience?
Random sample – Randomly choose a percentage of your mobile customers to
survey
Pros:
➢ Diversity of responses makes the data highly representative of your overall
audience
➢ Surveys can be prompted even if mobile analytic capabilities do not collect
information on customers and customer activity
Cons:
➢ Sample may be too broad to address narrow research questions that require a
high level of familiarity with the app
➢ Responses may be too generalized to uncover trends based on customers’
familiarity with the app, device used to access the app, etc.

WhoIsYourAudience?
New Users – prompted during the first hour or so of using the app
Pros:
➢ Allows you to collect information, unbiased by existing loyalty, that can be used to
improve the customer experience
Cons:
➢ Requesting customer information from first-time users may create a negative
initial experience with the app
Loyal Users – prompted the n-th time a customer opens the app or a few months
after installing and regularly using the app
Pros:
➢ Can be used to uncover which features loyal users find the most valuable / what is
bringing them back to your app
➢ Can be used to gather suggestions from those already familiar with those apps –
and customers can be further delighted if those suggestions are acted on
Cons:
➢ Responses may have an upward bias and have difficulty capturing equally
important negative experiences with the app

WhoIsYourAudience?
Event-Based Targeting – survey prompted at pre-specified ‘mobile moments,’ i.e.:
• The third time a customer uses the Search function
• The first time a customer shares content via the app
• After a customer updates to a new version
• After a customer uses a new/beta feature
After 3 Searches
After a customer
Reserved a Table
Pros:
➢ Can be used to address narrow, feature-specific
research questions
➢ Can be used to refine and beta test new versions
and rollouts
➢ Respondents inherently have familiarity with the
app event they’re providing feedback on
Cons:
➢ Hardest to integrate without third-party mobile
engagement solutions

MappingYourPlan
What are
you
looking to
learn?
What sort of
data
provides this
insight?
What questions will
provide the data you
need?
Who is
your
audience?
How will
you reach
them?
How can
this app
be
improved?
Mix of
quantitative
and
qualitative
How would you rate the
following features:
[rating scale]
Do you have any
suggestions for how we
could improve this app?
[open-ended response]
Returning
users who
have
opened
your app
3+ times
Prompt on
next log-in

BestPractices:MobileResearchDesign
When designing your mobile research instrument, do:
➢ Design with mobile in mind
➢ Keep questions brief and concise
➢ Make questions optional and allow customers to opt out at any time
➢ Aim to address your research objective with as few questions as possible
➢ Limit the number of options for multiple choices
➢ Break the questions up so that only one or two appear at a time
➢ Provide an ‘Other’ field with a textbox for fill-in answers to your multiple
choice questions if you suspect that some respondents may have answers
you had not previously considered
➢ Add an option for ‘Don’t Know’ or ‘Not Applicable’ for questions that some
respondents may not be able to answer
➢ Pre-test your survey internally to identify any weaknesses and ambiguity

BestPractices:MobileResearchDesign
And don’t:
➢ Create overlap in multiple choice responses. All responses should be
mutually exclusive
➢ Present rating scales with large matrices of options or questions ones that
require scrolling on a mobile screen
➢ Create vague responses that are open to the interpretation of the
respondent (i.e.: If asking about use frequency, give tangible options like
‘twice a week’ and ‘once a month’ rather than ‘often’ or ‘rarely’
➢ Frame questions in a way that leads the respondent or creates bias (i.e.:
“Why do you like this app?”)
➢ Request personal information at the start of the survey as this may lead to
lower response rates. If you need this information, make the questions
optional and move them to the end of the survey

BestPractices:SurveyIntegration
For best results integrating your research instrument within your app, we
recommend you:
➢ Use an in-app survey rather than directing mobile customers to a web survey so
as to not detract from the customer experience. If you are using a web survey, be
upfront about asking customers to leave the app for an external link.
➢ Use event-based targeting that isn’t intrusive. Don’t immediately ask new users to
take a survey, and only ask customers to take your survey once rather than asking
each time they load an event.
➢ Integrate the survey with your existing customer analytics to allow you to target
the responses against your audience segmentation to uncover trends based on
loyalty, device, etc. without having to ask customers to fill out additional
questions.

AdditionalResources
We hope this guide has helped kick start a conversation around why it’s important to
do mobile research and how to design an effective survey instrument.
For additional tips, we recommend SurveyMonkey’s Guide to Smart Survey Design:
http://www.slideshare.net/ROhl/surveymonkey-smart-survey-design-guide
As well as these related posts on the Apptentive blog:
➢ The Importance of Mobile Feedback
➢ Why Your Mobile App Needs In-App Surveys
➢ 5 Tips to Writing Effective Surveys for Mobile Apps

Mobile research best practices

More Related Content

What's hot (20)

Viewers also liked (10)

Similar to Mobile research best practices (20)

More from Apptentive (8)

Recently uploaded (20)

Mobile research best practices