What Do Users Really Think? Surveying Users About Your Help Content

WHAT DO USERS REALLY THINK?
Surveying Users About
Your Help Content
Patty Gale

WHAT WILL YOU LEARN TODAY?
Why should you conduct a survey?
What can you expect to learn? not learn?
How do you design the survey?
What gotchas should you avoid?
Where do you find users to participate?
How do you conduct the survey?
What do you do with all those results?
How do you create an action plan based on the survey feedback?
How can you share the results with stakeholders
in a way that they can digest it?
WHAT DO USERS REALLY THINK? #tkrytr #usersurvey 2

WHY SHOULD YOU CONDUCT A SURVEY?
Where do they go for learning content?
Do they use your learning content?
What do they think about your learning content?
Are they satisfied with your learning content?
• CSAT = Customer Satisfaction
Learn how you can better meet the needs of your users.

WHAT CAN YOU EXPECT TO LEARN? NOT LEARN?
Learn
• What people think they do
• How they feel about it
• Feedback about their most recent
experiences
Not learn
• What people actually do
• How they use it
• Feedback about past experiences
So why do it? Because you can still learn a lot from this information.

HOW DO YOU DESIGN THE SURVEY? TOOLS
http://www.capterra.com/survey-software/
• Easy to use
• Free trial/free edition
• Supports skip logic
• Supports many types of questions
• Survey management tools
• Analysis tools/features

HOW DO YOU DESIGN THE SURVEY?
Goals Questions Plan Peripherals Feedback Pilot

IDENTIFY GOALS
What’s the point you’re trying to prove,
or the question you’re trying
to answer?
Why do you want
to survey users?
What do you hope
to learn?
EXAMPLE:
Measure usage and satisfaction
with learning content
Identify areas and
opportunities
for improvement
Understand how we
can better support the
learning needs of our users

DESIGN QUESTIONS: CHOOSE A DEMOGRAPHIC
Who will you send the survey to? How will you distribute it?

DESIGN QUESTIONS: BIAS

DESIGN QUESTIONS: AVOID BIAS
We think our online help is really awesome.
How awesome do you think our online help is?
How would you rate your satisfaction with the XYZ online help?
    
Very Dissatisfied Neutral Satisfied Very
Dissatisfied Satisfied
https://www.surveymonkey.com/mp/survey-guidelines/

DESIGN QUESTIONS: SCREENING
Which product do you use the most?
Do you use the software at least weekly?
Which software version do you currently use?
rest of survey
A, B, C
Yes
2015, 2014
No
None of these
A prior version
A general question?
exit survey

DESIGN QUESTIONS: A GOAL FOR EACH
Every question must have a purpose.
If you are asking it just because you are curious about the answer, eliminate it.
WHAT DO USERS REALLY THINK? #tkrytr #usersurvey
When you need
information about
how to use XYZ,
how do you find the
information you
need?
Goals:
Understand where users instinctively
go when they need help.
Consider whether their favorite
resources have characteristics or
information types that we can
emulate to improve the quality of
our learning content.
12

DESIGN QUESTIONS: DEMOGRAPHICS
Include a few demographic questions that will allow you to slice and dice
the resulting data to compare responses from different groups of users.
Approximately how long have you
used or worked with XYZ software?
• Less than 6 months
• 6 to 12 months
• 1 to 3 years
• 4 to 5 years
• 6 to 10 years
• More than 10 years
When you need information
about how to use XYZ,
how do you find the
information you need?
• Search online
• Visit expert blog
• Use product help
• Contact customer support
• Ask co-worker
• Other (please specify)
13

Goals
Identify
goals for the
survey
Questions
Design questions
to meet goals
Plan
Make a data
analysis
plan
Peripherals Feedback Pilot

MAKE A DATA ANALYSIS PLAN
• Before you send out the survey
• How will you analyze the
results from each question?
• How will you use the resulting
information?

DATA ANALYSIS PLAN: EXAMPLE 1
Goal:
This question is intended to help us set priorities on
the types of content to develop going forward to
meet our users' needs.
Choices include the following:
• Troubleshooting
• In-depth information about how XYZ works
• Step-by-step instructions
• Video tutorials
• Detailed workflows between multiple products
• Examples
• Other (please specify)
In the XYZ learning content, what types of information would you like to see more of?
(choose 3)
Analysis:
Determine the top 3 requested types of information.
Do these reflect current initiatives?
• If so, it's confirmation that we are moving in the right
direction. Be sure to perform follow-up user testing
and/or surveys to make sure we hit the mark.
• If not, consider whether we should place an emphasis
on these types of information for our sprint work.
Examine responses for "Other." Are they requesting
any types of information that we haven't considered?
If so, investigate further.

DATA ANALYSIS PLAN: EXAMPLE 2
What can we do to improve your experience with XYZ learning content?
(open text response)
Analysis:
Look at individual responses. What specific suggestions can we implement?
Make user stories to add to a future sprint to make these improvements.
Look for overall themes:
• What do multiple respondents ask for?
• What areas of learning content seem to cause the most angst?
• What larger improvements can we make that might help to reduce pain/frustration
for a large group of users?
Identify these themes and make an action plan to implement a solution.

Goals
Identify
goals for the
survey
Questions
Design questions
to meet goals
Plan
Make a
data
analysis plan
Peripherals Feedback Pilot

CREATE SURVEY PERIPHERALS
End of survey
Invitation Introduction
19

INTRODUCTION
• Message before the survey
questions
• Short and sweet
• What is it? (a survey)
• What is the reason for the survey?
• How long will it take?
• Signed by: the name and title of
the person sponsoring the survey.
(a specific individual,
not just a group or division)
You are invited to participate in the XYZ Learning Content
Survey.
By participating, you are helping us to:
• Understand your experiences with XYZ learning content and
how you use it.
• Get your feedback on how we can improve existing learning
content.
• Set priorities for future learning content development.
The survey should take about 10-15 minutes of your time. If you
cannot complete the survey in one session, you can stop at any
time and your answers will be saved. To continue where you left
off, click the link in the email invitation.
Thank you for your feedback.
Sincerely,
Somebody Important
Director, User Experience
XYZ Corp.

INVITATION
• Short and sweet
• Repeats a lot of the
information in the survey intro
• Provides the survey link
• Signed by: the name of the person
who is sending out the invitation,
and the group they are working
with for the survey
Dear <name>,
As an XYZ user, what is helpful to you when using learning
resources for XYZ? What learning content is most relevant to
you?
The XYZ Learning Content Team would like to hear from you. By
participating, you are helping us to:
• Understand your experiences with XYZ learning content and
how you use it.
• Get your feedback on how we can improve existing learning
content.
• Set priorities for future learning content development.
The survey should take about 10-15 minutes of your time. If
you cannot complete the survey in one session, you can stop at
any time and your answers will be saved.
Please click the link below to provide your feedback.
Thank you.
<Survey link here>WHAT DO USERS REALLY THINK? #tkrytr #usersurvey 21

THANK YOU
• Message after survey questions
• Short and sweet
• Give them the chance to provide
final feedback
• Provide an email in case they’d
rather reach out in person or they
think of something after submitting
the survey
If you have additional comments about the XYZ learning
content or questions about this survey, enter them below,
or send email to learning-content@xyz.com.
Do you have additional feedback about XYZ learning
content?
[ Submit ]
Thank you for participating in the survey. Your feedback is
important to us.

Goals
Identify goals
for the survey
Questions
Design questions
to meet goals
Plan
Make a data
analysis plan
Peripherals
Write
the invitation,
intro text,
thank you text
Feedback
Show it to others
for feedback
Pilot

GET FEEDBACK
Share
• Show the completed survey (including peripherals) to colleagues
• Especially any with experience in user research and user
experience
Review
• Ask them to read through it carefully and provide feedback
• Any feedback is welcome: wording, questions, length, focus, etc
Revise
• Incorporate their feedback
• Rinse and repeat
24

Goals
Identify goals
for the survey
Questions
Design questions
to meet goals
Plan
Make a data
analysis plan
Peripherals
Write
the invitation,
intro text,
thank you text
Feedback
Show it to others
for feedback
Pilot
Send it to
a small group
of internal
users to
identify and
fix issues

PERFORM A PILOT TEST
Goal: Identify and correct issues with the survey before it goes to customers.
Invite 3-5 internal colleagues to take the survey.
• Who? Folks who can represent the user to some extent. QA? UX? PM?
• Send them the invitation with the link, and have them complete the survey.
Ask them for direct feedback:
• Are the invitation and introduction clear?
• Are the questions clear? Did any cause confusion?
• Is the survey too long?
• Did they encounter any issues? Or think of questions as they took the survey?
Revise the survey, and have a few more folks take it. Rinse and repeat.
26

WHAT GOTCHAS SHOULD YOU AVOID?
• Qualified respondents? >> screening questions
• Wording of questions
• Number of questions (survey fatigue)
• Order of questions
• Timing of the survey (summer vacation, winter holidays)
• Duration of survey (2-3 weeks)
• If no NDAs, do not ask about or show content that is not yet released
• If your organization conducts many surveys, avoid survey fatigue

WHERE DO YOU FIND USERS TO PARTICIPATE?
SurveyMonkey Audience
 “Access millions of highly qualified, on-demand survey takers. Choose from 30+ criteria to reach the
people you need. Get results in 2 days and watch data appear in real time”
 Better suited to consumer goods and services
Talk to Product Management, Customer Support. Who is actively using the software?
Marketing >> customer list?
Get a list of beta testers. (may be under NDA)
Goal: 100+ responses.
How many invitations will it take to achieve that goal?

HOW DO YOU CONDUCT THE SURVEY?
Plan the
timing
and
duration
Send
invitations
Wait
and
watch
Send
reminders
after
one week
Close
the survey
29

WHAT DO YOU DO WITH ALL THOSE RESULTS?
Quantitative results
• multiple choice, Y/N, rating scale
• use graphs & charts to easily
visualize the results
Qualitative results
• open-text response
• analyze text
30

QUANTITATIVE: CHARTS AND GRAPHS

QUALITATIVE: AFFINITY DIAGRAMMING

QUALITATIVE: IMPORTANCE-DIFFICULTY MATRIX
Important
Difficult (time, cost, culture)
Not Important
Easy
Luxury Strategic
Quick winsTargets
Resolve differing opinions
Prioritize ideas quickly
Leads easily to an action plan
33

HOW DO YOU CREATE AN ACTION PLAN BASED
ON THE SURVEY FEEDBACK?
• Look for themes.
• Quantitative results: What did you learn?
• Qualitative results: What patterns emerge?
• What do you have control over?
• Make a plan to implement improvements/changes.
• Get management buy-in.
• Implement the plan.
• What don’t you have control over?
• Share your survey results with the responsible group.
• Suggest potential changes they could make to improve
the user’s learning experience.

HOW CAN YOU SHARE THE RESULTS?
…with stakeholders in a way that they can digest it?
• Boil it down. What is the main point that you want to share?
• Provide supporting evidence: one or two charts, supporting quotes from
open-text responses.
• Summarize your action plan: what will you do in light of these results?
• AVOID death-by-PowerPoint

HOW CAN YOU SHARE THE RESULTS?
Boil it down
to the main points
Provide supporting
evidence
Summarize your
action plan
36

NOW YOU KNOW WHAT USERS THINK
and you have a better understanding
of how to create learning content
that supports their needs

RESOURCES
Basics of creating and running a useful online survey:
http://uxmastery.com/better-user-research-through-surveys/
Designing questions: https://www.surveymonkey.com/mp/survey-guidelines/
Survey tools: http://www.capterra.com/survey-software/
Sharing research results:
• https://medium.com/%40FBResearch/beyond-bullet-points-four-creative-
ways-to-share-research-c10fa047f025#.sayukl640
• http://www.userfocus.co.uk/articles/dashboard.html
• http://www.userexperience.co.nz/2015/12/visualising-design-research/
• http://annkemery.com/qual-dataviz/

ATTRIBUTIONS FOR IMAGES
From the Noun Project (CC-by-3.0)
• Show-and-Tell by Aha-Soft
• Moods: Excellent, Good, Regular, Bad,
Terrible by Rflor
• Mom and Boy by Fx Pedron
• Toy Robot by Oliviu Stoian
• Map by Alen Krummenacher
• Analysis, Error by Chameleon Design
• Flow chart by Jhun Capaya
• Invitation by Creative Stall
• Add conversation, Successful conversation by
Oliviu Stoian
• Users by Wilson Joseph
• Group by Felix Westphal
• Demographics by Fiona OM
• Chart by Pham Thi Dieu Linh
• Documents by Eugen Belyakoff
• Boil by Antistatique
• Bar Chart by Pham Thi Dieu Linh
• Pages by Eugen Belyakoff
• Reports by Joao Marcelo Ribiero
• Calendar by Ed Harrison
• Presentation by Garrett Knoll
• Question by Fred Selbert
Public domain:
• Meeting (qualitative results)
• Confused
Other:
• Affinity diagramming photos by Patty Gale
• Quantitative charts by SurveyMonkey
39

Patty Gale
patricia.gale@autodesk.com
LinkedIn: patty-gale-9774461
Twitter @tkrytr
40

SURVEY QUESTION:
SOURCES OF INFORMATION
When you need information about how to use XYZ,
how do you find the information you need? (Choose up to 3 methods.)
 I search online (Google, Bing, etc.).
 I visit the blog/social media of my favorite XYZ expert.
 I visit a XYZ user forum.
 I access product help (for example, by pressing F1 or a Help button in XYZ).
 I contact XYZ Customer Support.
 I ask my co-worker for help or use an internal resource.
 Other (please specify)
41

QUESTION: WHEN DO YOU NEED LEARNING
CONTENT?
When do you use the learning content produced by XYZ Inc?
(Check all that apply.)
 When I can't remember how to complete a task
 When I don't know what something on the screen means
 When my co-workers are not available to answer my questions
 When I want to find a quicker way to do my work
 When something goes wrong and I don't know how to fix it
 When I need to research features so I can teach others
 When I am teaching others
42

In the XYZ learning content, what types of information
would you like to see more of? (Choose up to 3.)
PRIORITIES FOR FUTURE LEARNING
CONTENT
 Troubleshooting
 In-depth information about how XYZ works
 Step-by-step instructions
 Video tutorials
 Detailed workflows between multiple products
 Examples
43

What Do Users Really Think? Surveying Users About Your Help Content

More Related Content

What's hot (20)

Viewers also liked (7)

Similar to What Do Users Really Think? Surveying Users About Your Help Content (20)

More from patricia_gale (9)

Recently uploaded (20)

What Do Users Really Think? Surveying Users About Your Help Content