Capturing and Analyzing Qualitative Data in Surveys

Capturing & Analyzing
Qualitative Data in
Surveys
Karen Carleton, MEd, MS
www.performcorp.ca
International Institute of Business Analysts
April 23, 2012

© Karen Carleton 2012

Objectives
1. Identify organizational context (PI)
2. Overview of survey design
3. Data collection & analysis best practices
(Qualitative data focus)
Exercises:
 Question critique
 Question writing practice (scenarios)
 Survey plan (real)
 Review

An ounce of
analysis is worth
a pound of
objectives
–Joe Harless

BA Competencies

 Analytical thinking
 Problem-solving
 Organizational learning
 Systems thinking

Source: Business Analysis Book of Knowledge

Organizational
Performance Improvement

RSVP Approach
Results-oriented
Systems thinking
Value-add
Partnerships focus

Systematic Solution Design

ADDIE e.g. Frontline hours extended
Analyze unintended problems elsewhere
Design
Develop
Implement
Evaluate

Organizational Research

Process improvement
Re-engineering
Organizational change
Software development

Others?

Source: Business Analysis Books of Knowledge

Surveys

Think-Pair-Share

Why might you
conduct a survey in
your organization?

Survey Objectives

Problems
 Determine causes, impacts, optimal solution(s) &
solution performance

Opportunities
 Improvement (efficiency/effectiveness), define
business need or requirements

Stakeholders
 Elicit stakeholder needs/concerns, solution req’ts


METHOD DESCRIPTOR
Survey/ Collects topical data from a representative
Questionnaire population
Focus group Discussion-based group interview, noting
ideas, themes & non-verbals
Interview One-on-one questioning (structured to
unstructured)
Document Background info, context (e.g. job
review descriptions, stats, annual report)

Observations Note details observed – situations/
behaviours (e.g. procedures)

Surveys/Questionnaires

Elicitation Techniques (methods)
 Interviews, case studies/scenarios,
interviews, document analysis, focus
groups, questionnaires/surveys

Attitudes, opinions, recommendations,
experiences



More than 1 method (technique)

Triangulate


More than 1
stakeholder
group

Triangulate

Example: BPI

Target Audience Example

 Select participants
 useful/relevant input

 Representative
Sample
 Determine precision
needed

Best Practices

 Informed consent
 Purpose,
confidentiality,
results
 Organized,
consistent
 Know audience &
resources

Survey Types

In-person

Phone

Web-based

Which type
Survey Types & why?
 Flexible, interactive
 Clarify, probe
 Body language/non-verbals

 Flexible, interactive
 Clarify, probe; no travel
 Voice tone, intonation

 Easy for large respondent group
 Cost-effective, convenient

Considerations
Target
Objective Audience

Resources Data Types

Surveys

 Objective
 Target audience(s)
 Location(s)
 Resources ($, time, help)
 Data needed (quantitative &/or
qualitative)

Examples?

Intro. Message

Objective/Purpose
Thank
Results
Confidential;
anonymous
Voluntary

Staff Satisfaction Survey
At ABC Company employee feedback is important to
us. We would appreciate it if you would take 5-10
minutes to complete this survey by Monday May 7,
2012, 2012. All responses are anonymous and will be
kept confidential.
Your participation is voluntary and you may skip any
questions or discontinue at any point. By June 15, 2012
we will post the aggregate results and action items on
the intranet. Thanks in advance for sharing your
thoughts and opinions about how ABC Company could
be a better workplace!

E-Surveys

Advantages Disadvantages
 Low cost, timely  Computer literate &
 Large group, dispersed access
 Data aggregation &  No follow-up/
storage (no data entry) probing
 Anonymity  No intonation,
gestures
 Format ease
 Little contact
 Skip logic, progress bar
 Mostly US servers

Quantitative Data

 Numerical (“objective”) data
 Measured, statistical, stable study
 Length, height, volume, weight, speed, time,
temperature, sound levels, cost, age range, #
users or groups, %, etc.

Qualitative Data

 Descriptive (“subjective”) data;
not traditionally measured
 Textual, flexible study
 Colour, texture, appearance, beauty, smell,
taste, etc.

Latte Example
Quantitative Qualitative
 12 ounce mug  Medium; see-
 7 “ mug height through mug
 Serving  Steaming hot
temperature 150ºF  Pricey drink
 Cost: $4.95  Robust aroma
 Frothy;
Delicious

Quantitative Data Analysis

 Trends
 Average (mean),
highest & lowest
scores
 Generalizable
sample -
Statistically
significant &
representative

Qualitative Data Analysis
 Descriptive responses
 Feelings, beliefs,
likes/dislikes, thoughts,
personal opinions,
comments
 Patterns/Themes,
Counts

Qualitative Data

Recommendation:
Couple quantitative
(numerical) & qualitative
(descriptive) data

“Confidence”, “Rigor”,
“Reliability”, “Validity”,
“Consistency”, “Truth”

Qualitative Data (open-ended
questions) Limit use!
Advantages Disadvantages
 Personally  Non-response – more
meaningful, cultural time/effort to do
 Deeper questions (higher skip rate)
 Rich, complex human  No follow up/clarifying
details explained  Longer to analyze
 Ideas not (more resources –
anticipated time, people)


Response Rate

# Surveys completed = Response rate
# Participants contacted

 Sufficient numbers important
 Too little data= may not have substantial
info.

Example: client survey

Increase Response Rates

1. User-friendly, brief
2. Rewards-Cost-Trust
 R=Value, positive regard, thanks, token
WIIFM (incentive)
 C=short, easy
 T=important, authority, confidentiality
3. Reminders
Example: BSU faculty/staff, & students

Questions

 Objectives –
ask targeted
questions
 Clear, concise

Closed Questions (Quantitative)
 Rating/Matrix (4, 5 or 7 point)
 Ranking, ordinal (1-4) – order
 Multiple choice (one or more responses)
 Likert scale (Agree…), descriptors

Open Questions (Qualitative)

 Fill-in-the-blank
(# or word)
 Text box
(comment or
paragraph)

Example: Staff Survey
“career development”

Question Critique
Under what circumstances Negative phrasing
do you not submit your
weekly project hours?
 Overtime still in progress
 Unknown factors
 NA  Other

Do you like our Double-barreled
Edmonton and Calgary
branches?
 Yes  No

Question Critique
What would make the Acronym or
EFAP better? jargon; unclear

If you are working on a Confusing/
late project and a team complex format
member is away, do you:
a) wait
b) continue
c) ask your supervisor for
advice

Question Critique
How fair was your Sensitive
performance bonus this year?
Strongly agree Agree Neutral Disagree Strongly disagree
5 4 3 2 1

Do you like the coffee? “Apple pie
Yes  No effect”
Do you like the washrooms?
Yes  No
Do you like the parking? Yes  No
Do you like your boss? Yes  No

Question Critique
How many copies did Lack of alternate
you make with the choice when
area’s photocopier last needed: “N/A”,
week? “no opinion” or
a) Less than 5 “prefer not to
b) Less than 20 answer”, ”don’t
c) Less than 50 know”
d) Less than 100
e) 100 or more

Question Critique

I generally feel informed about changes
that affect me at ABC Company.
1 2 3 4 5
Not at all Very
Informed Informed

Unclear language & descriptors

Question Critique

I can see the link between my work and
ABC Company’s objectives and
mission.
1 2 3 4 5
Not at all Very

Double-barreled question,
Presumes understanding (2),
Descriptors don’t fit.

Question Critique

Do you feel the efforts of the Social
Committee contribute to your
enjoyment in working at ABC
Company?
1 2 3 4 5
Not at all Very
Leading question: presumes
knowledge & enjoyment.
Descriptors don’t fit.

Question Critique
My working conditions are good.
1 2 3 4 5
Strongly Strongly
Disagree Agree

My relationship with my manager is good.
1 2 3 4 5
Strongly Strongly
Disagree Agree
Vague adjective

Question Critique

Is there anything else you would like to
mention that you feel could help improve
staff satisfaction at ABC Company?

Wordy

Question Tips
At start, interesting question
Easier before difficult ones
At end, personal & demographic
Consider analysis
Have others review
Pretest

Why?

Question Writing Practice
Scenario pairs:
1. Decide survey Objective
2. Choose a fitting Distribution Method
(e.g. phone, in-person, e-survey/web-
based)
3. Create 3 Questions Time: 20 minutes
4. Question Review (with another group)

Time: 5 minutes

Data Analysis

Quantitative
 Total responses,
frequency counts,
distribution, averages

Qualitative
 Pre-determined or
emergent categories

Qualitative
Triangulation

1. Cross-check
data accuracy
 Validate, verify
1. Other method(s)
2. Other
stakeholder(s)

e.g. survey employees,
interview supervisor

Qualitative Data Analysis
2. Constant Comparison
Method
 Compare & contrast text
items
 Sort, sift, group, match
 Categorize, order,
sub-divide

“Raw” “Cooked” Data

Qualitative Data
Analysis
3. Read-Reread Method
 Iterative review
 Make sense of findings
(determine meaning)
 Underline, sort, count,
code, highlight data
 Patterns/picture(s)
emerge

Qualitative Data
Analysis
4. Assign Codes
 Colours, numbers,
symbols, categories,
abbreviations
 e.g. F (female)
 e.g. MROLE
(manager role)

Survey Plan
1. Set objective & 5. Pre-test survey
target audience 6. Administer survey
2. Choose distribution 7. Analyze results
method 8. Communicate &
3. Create questions address results
4. Question
review/test Practice with
first 3 steps

Survey Plan (real)
1. Objective
2. Target audience(s)
3. Survey distribution method
4. Additional research method(s) (*to verify
survey data)
5. Draft 1 clear, open-ended question

*2-4 pairs to share plan with group.
Time: 15 minutes (pairs)

Communicating Results

 Honesty about
findings
 Timely results
 4-6 weeks
 = TRUST
 Act on findings
or what
happens?

Review
1. One BA competency involved in survey
analytical or systems
design is ____________________ thinking.
2. Another organizational research method you
could use to verify your findings’ accuracy is
Interviews, focus groups,
___________________________________
observations, secondary data
___________________________________.
3. By using more than one method or
riangulate
stakeholder group you can T____________
your results to validate them.

Review

4. Surveys/Questionnaires are considered an
licitation
E_________ technique by BABOK.
5. Knowing the characteristics of your
target audience
_______________ is important for survey
design.
6. Advantages of E-surveys are that they are:
cost-effective, good for dispersed
______________________________________
populations, and ease of analysis
_______________________________.

Review

7. Numerical or statistical data is
quantitative
____________, while descriptive data with rich
qualitative
detail is ___________.
8. Qualitative data analysis involves seeking
patterns/themes
________________ in the text, by using the
re-read constant comparison
read-________ or __________________
methods.
9. Accurate qualitative data is described as
having confidence, rigor, validity, consistency
_______________________________.

Summary

 Know your audience, objective & resources
 Consider question types & analysis
 Triangulate (*more than one method, &
stakeholder group)
 Follow best practices
 Empathize with your respondents

Questions or comments?

Extras & Presenter Info
1. Write #1 YES on back of your business card for
the FREE “Top Ten Reasons Surveys Fail”

2. Write #2 YES on back of your business card for a
FREE Strategy Session.

Karen Carleton, MEd, MS
Performance Solutions Corp.
 Survey design & consulting
 Needs analysis & evaluation
www.performcorp.ca Email: Karen@performcorp.ca

References
 Business Analysts’ Book of Knowledge (BABOK)
 Dillman, D.A. (2000). Mail and Internet surveys: The
total design method (2nd ed.). New York: Wiley.
 Greatbrook. Statistical Confidence. Retrieved April 18,
2012 from,
http://greatbrook.com/survey_statistical_confidence.htm
 Inquisite survey software, http://www.inquisite.com/
 Qualtrics survey software, http://www.qualtrics.com/
 Survey Monkey survey software,
http://www.surveymonkey.com/
 FREE account, up to 200 respondents. “How to” downloads.

Capturing and Analyzing Qualitative Data in Surveys

More Related Content

What's hot (20)

Similar to Capturing and Analyzing Qualitative Data in Surveys (20)

More from Performance Solutions Corp. (7)

Recently uploaded (20)

Capturing and Analyzing Qualitative Data in Surveys