How to Benchmark Your Online Customer Experience Against Competition

How to Benchmark Your Online Customer
Experience Against Competition
1

2
Conducting Remote Unmoderated Usability Studies: Part 1
Ann Rochanayon
Sr. Director UX/CX Research
UserZoom
arochanayon@userzoom.com
Speaker
Meet the Speakers
Elijah Degen
VP, Head of Digital Marketing Practice
Interakt
elijahd@interaktco.com
Speaker

3
The #1 Platform for Agile Usability Testing
& User Experience Analytics
Our all-in-one enterprise solution makes research & testing
cost-effective, agile and easy

4
Quick Housekeeping
• Chat box is available if you have any questions
• There will be time for Q&A at the end
• We will be recording the webinar for future viewing
• All attendees will receive a copy of the slides/recording
Continue the discussion using: #uzwebinar
Continue the discussion using #uzwebinar

5
UserZoom Retail Benchmark | December 2014
Contents
• How CE/UX Benchmarks Add Value to Your Brand
• About Remote Unmoderated User Testing
• Case Study: Retail Competitive Benchmark Study
 Study Methodology
 Participant Profile
 Summary
 Brand, Conversion, and NPS Findings

6
How can CE/UX benchmarks
add to value to your brand?

Online User experience – the defining moment
7
The content and quality of each individual visitor’s interaction — the user
experience — collectively defines your digital brand, communicates your value
proposition, and creates the foundation for future brand interactions — positive
and negative.
Perceived
Credibility
Profitability
Intent to
Return
Intent to
Purchase
User
Satisfaction
Word of
Mouth
A Positive
Experience
A Negative
Experience
Expectations User Experience

8
Lots of macro benchmarks are out there…
• Traffic, user demographics, media spend, etc…
Why are there few benchmarks for CE/UX?
• Customer Experience & User Experience have varying
definitions between companies
• Usability benchmarks are generally custom
• Competitors vary depending on who you ask
• Info from UX benchmarking is so actionable, it is rarely
shared

Where can UX benchmarking impact decisions?
9
Area of Discovery Example value
Best Practices in service/UX design Leverage competitors findings and save
costs
Service/UX Design Decisions Justify investments/costs via VOC
Or test prototypes
Know your customer Know what keeps them from converting
Quantify to customer feedback Give hard numbers to show impact.
Example Question:
For those who considered your brand,
what are the top 5 reasons why not
converting?
Online Brand Impact Help brand marketers understand the
brand's impact in online experience
Eg Brand Perception can impact
conversion as much as online task
completion.

10
What can be measured?
Web design components
• Navigation, content, site performance, interactive
elements.
• Example Metrics: Task Completion, Ease of Use, Time
on Task, etc.
Brand components
• Voice of Customer, Brand affinity/likeability, Attitudes
toward Brand
• Example Metrics: NPS, CSAT, Brand Attributes Matrices

11
What is needed to benchmark properly?
Area of Discovery Example value
Correct tools • cross device
• website independent
• In-app
• Video & audio & clicks
Correct Audiences • Target Audience
• Counter Audiences
eg Lost Opportunity Audience, Control Group
Correct Proctor / Partner • Impartial = Let the numbers speak for
themselves
• Specialists w/ Broad Experience
Correct Type of Testing • Moderated In-Person
(powerful exploratory / directional tool, but EXPENSIVE)
• Unmoderated Remote Testing
(great deep dive analysis and very cost scalable globally)

12
Remote Unmoderated User Testing
• ‘In-the-wild’ testing (natural context)
• Geographic representation
• No moderation needed
• Software collects data,
stores securely in the cloud,
accessible real time
•Test any web-based UI, live
or prototype
• Output: Usability + Behavior
+ Video & Audio +
Satisfaction+NPS+Survey
Feedback
• Fast data collection
• Cost efficient

13
Methodology
• Goals
• Determine which retail site presents the best tablet shopping experience
• Assess the impact of the tablet shopping experience on brand perception, Net
Promoter Score and conversion
• Sites evaluated
• Participants tested one of four sites in Nov - Dec 2014 on a tablet
• N=202
• Ages 18+
• Shopped for clothing and accessories online at least once per year

14
Methodology: Study Design
• Between-subjects design
• N=202 (~50 per site)
Screener
Introductory Questions
Macy’s
1. Find a jacket
2. Return policy
3. Checkout
Nordstrom
1. Find a jacket
2. Return policy
3. Checkout
Urban Outfitter
1. Find a jacket
2. Return policy
3. Checkout
Lands’ End
1. Find a jacket
2. Return policy
3. Checkout
Final Questions

15
Participant Profile: Purchase experience
Please indicate your gender.
84%
16%
Female
Male
5%
42%
17%
14%
21%
18 - 25
26 - 35
36 - 45
46 - 55
Over
55
Please indicate your age range.
Which of the following best describes how frequently you
purchase clothes and accessories online?
15%
39%
38%
8%
Weekly
Monthly
Several times per year
Once per year
N=202

16
Participant Profile: Sites Browse / Purchased
• Kohl’s and Macy’s were the top two sites participants visited to purchase
clothing and accessories
Which of the following websites have you browsed for clothes and accessories using your tablet? / Which of the
following websites have you purchased clothes and accessories using your tablet? [Check all that apply]
20%
11%
14%
14%
34%
13%
62%
21%
14%
52%
26%
6%
7%
10%
18%
5%
38%
9%
6%
37%
10%
5%
Aeropostle
Banana Republic
Dillard’s
GAP
J. Crew
Kohl’s
L.L.Bean
Land’sEnd
Macy’s
Nordstrom
Urban Outfitters
Browsed Purchased N=202

18
User Experience Rankings
User Experience
Score*
79
72
70
55
1
2
3
4
To assess the overall user experience ranking of the sites, UserZoom captured usability
metrics for three key tasks on each tablet site. Participants were asked to find an item
meeting specific criteria, find return policy information and asked to evaluate the checkout
process. The usability metrics captured include key performance indicators such as task
effectiveness, ease of use, and clarity of information. The rankings were derived according
to how each site performed on each measure.
*Score based out of 100 possible points

19
Summary: Task Effectiveness
• Participants were most successful in completing the study tasks on the Lands’
End site
• Urban Outfitters had the lowest overall task success rates
96% 98%
98% 98% 98%
88% 86% 90%
94%
79%
69%
85%
Task 1: Find a Jacket Task 2: Return Policy Task 3: Checkout
Macy's N=50 Nordstrom N=49 Urban Outfitters N=52 Lands' End N=51

20
Summary: Task Ease of Use
• Lands’ End performed relatively well overall on task ease of use
• Urban Outfitters had the low ease of use ratings, consistent with the lower task
success rates
72%
48%
68%
65% 67%
78%
37% 38%
62%
69%
61%
86%
Task 1: Find a Jacket Task 2: Return Policy Task 3: Checkout
Macy's N=50 Nordstrom N=49 Urban Outfitters N=52 Lands' End N=51

21
Summary: Lands’ End
• Lands’ End led all sites with
the highest ratings in the
usability metrics
• Nearly all participants
succeeded in all three tasks,
more than any other site
• Lands’ End also had relatively
high ease of use ratings and
was rated highest on post-study
brand perception
• Participants found the Lands’
End site fast, simple, easy to
use and understand; Lands’
End also provided clear return
policy information
• Although Lands’ End rated highest on task success and fewest issues encountered
for the Return Policy task, one area of improvement to consider would be cleaning up
the look and feel / organization of the Return policy page

22
Summary: Lands’ End
• Study participants appreciated the fast, simple, easy to navigation with clear
information available on Lands’ End
Simple and easy to use
“Very easy to navigate”
“Overall a very easy pleasant experience”
“Very easy, big selection”
“A lot of ways to narrow your search. Easy
checkout process”
“Everything was so simple and easy to do.”
Clear Information, Easy to Understand
“Easy to navigate site. Information clear and easy
to understand. No issues.”
“The site was very easy to navigate, all necessary
information was given, pictures and descriptions
of merchandise was clear”
“Very easy to maneuver around, easy to
understand return policy”
Fast
“Easy and fast to find what I was looking for”
“Your website does not lag at all which made it an
enjoyable experience.”

23
Summary: Nordstrom
• Nordstrom edged out Macy’s for second place in the user experience rankings
• A vast majority of Nordstrom participants succeeded in all tasks with 88% or more succeeding
• Task ease of use were fair with 65% of the participants or higher finding the site easy to use
while performing the assigned tasks
• Participants overall found the Nordstrom site easy to navigate and several mentioned the
helpfulness of the filters
• However, a sizeable percentage of participants encountered technical issues on the Nordstrom
site with 27% reporting site slowness and 24% had problems with pages jumping around when
looking for a product
• Another opportunity for improvement for Nordstrom would be providing clear return shipping
cost information; 27% reported frustration not understanding the cost to return an item by mail

24
Summary: Nordstrom
• The ease of navigation and product filters were cited as being the most helpful
aspects of the Nordstrom site
Easy to Navigate
“It was easy to use their website and find the
information I needed to make a purchase”
“Very easy to navigate and checkout...also great
return policy with free returns”
“How relatively easy it was to make purchases
and to be able to ship those items by mail.”
“Easy to navigate. Inviting layout. Easy to add
items to shopping bag and to checkout.”
Easy-to-use Filters
“The website was set up for easy purchases. And
the filters made my shopping very easy and fast.
Very impressed!”
“The whole thing was very smooth and easy to
understand. The filters, the check out, customer
service- it’s all very accessible.”
“The site was pretty easy to navigate with lots of
filtering options. Most of the information I looked
for was easily found”
“Easy to use the filters to narrow a search.”

25
Summary: Macy’s
• Macy’s finished in third place based on the user experience rankings
• Nearly all participants in succeeded in finding a jacket meeting specific criteria and
finding the return policy while 86% succeeded with the checkout task
• Despite the high success rates in finding the return policy, only 48% found the task easy
with 36% reporting difficulty understanding the shipping cost and 18% struggling to find
return policy information
• 68% found it easy to check out on the Macy’s site; technical issues with the site were
the primary problems encountered
• 16% mentioned the pages were too slow during checkout and 12% had problems with
the page jumping around when clicking

26
Summary: Macy’s
• Participants found Macy’s easy to navigate, but also mentioned that the site
was fast and offered a good tablet experience
Easy to Navigate
“Fast. Easy to find things and search”
“Ease of the website, clear self explanatory instructions
on how to check out and add things to my bag”
“Site was very easy to navigate, simple links and the
filter/sort feature was very fast.”
Fast Site
“Many products, fast loading”
“The site was easy to navigate, loading time was fast and
almost everything was very easy to locate.”
“Very easy and clear to navigate. Loaded quickly and
responded in a good amount of time.”
Good Tablet Site
“It worked really well on my tablet. I do most of my online
shopping on a tablet and some sites work better then
other. I would say this one was easy to use its not a site
that I would have to switch to another computer to use.”
“I like the mobile version better than desktop”

27
Summary: Urban Outfitters
• Urban Outfitters lagged behind the
other sites in fourth place
• Participants were least successful
finding a jacket, finding return policy
information and checking out
• Ease of use ratings were also lower
than the other sites
• Only 37% found it easy to find a jacket
with 23% struggling with the amount of
scrolling, 17% struggling with the filters
and 17% having problems narrowing
down results among other issues
• With the return policy, 37% had
problems finding it and another 13% felt
there was too much scrolling required

28
Summary: Urban Outfitters
• Participants had greater difficulty navigating Urban Outfitters due to the
scrolling required along with technical problems with the site
Difficult to Navigate
“Everything so complicated to find. Too much scrolling to
find an item. Info text too small to read”
“Not easy to navigate thru to see items. Need more
groupings etc., men’s coats, women’s coats, shoes men’s or
women’s.”
“It is hard to maneuver to categories because the pictures
are so big for the categories.”
“Images were too big, I had to scroll around too much”
Unclear Icons
“Also need to identify the BAG icon.”
“Icons are not easily identifiable.”
Technical Issues
“The checkout kept asking for my shipping address and
then would reload the same page. I did this 4 times.”
“The page did not fully load”
“The page would not bring up the items; the page kept going
blank”

29
Brand, Conversion, and NPS
Findings

30
Brand Perception
• After interacting with the site, Lands’ End had the biggest increase in positive
brand perception
62%
57%
23%
18%
62%
69%
44%
76%
N=37 N=50 N=37 N=49 N=31 N=52 N=39 N=51
Macy's Nordstroms Urban Outfitters Lands' End
Pre-site evaluation Post-site evaluation
Site Delta
Macy’s 0%
Nordstrom 14%
Urban Outfitters 21%
Lands’ End 58%
* Selected 6-7 on a 7-point scale where 1 = Negative and 7 = Positive

31
Call to Action
• Participants were most likely to make a purchase from Macy’s or Nordstrom
and least likely to to make a purchase on the Urban Outfitters tablet site
52%
Based only on your experience today, how likely are you to
make a purchase on [SITE] on your tablet in the future?
47%
15%
39%
38%
35%
43%
57%
10%
18%
42%
4%
Macy's N=50 Nordstrom N=49 Urban Outfitters
N=52
Unlikely (1-2)
Neutral (3-5)
Likely (6-7)
Lands End N=51

32
Net Promoter Score
• Participants were most likely to recommend Macy’s, followed by Nordstrom
• Urban Outfitters had far more detractors than promoters with a -56% NPS
NPS
10%
-6%
-56%
- 23%
How likely are you to recommend [SITE] to a friend or
36%
43%
colleague?
69%
45%
18%
20%
17%
33%
46%
37%
14%
22%
Macy's N=50
Nordstrom N=49
Urban Outfitters N=52
Lands End N=51
Detractor (0-6) Passive (7-8) Promoter (9-10)

33
Post Site Interaction Brand Attributes*
• Nordstrom was associated with ‘Customer-oriented’ and ‘High quality,’ while
Macy’s was most associated with ‘Value’ and ‘Variety’
42%
10%
51%
54%
37%
40%
52%
24%
63%
18%
41%
19%
23%
40%
12%
35%
8%
51%
31%
33%
Customer-oriented
Exclusive
High quality
Value
Variety
Macy's Post N=50
Nordstrom Post N=49
Urban Outfitters Post N=52
Lands End Post N=51
Based only on your experience today, which of the following
attributes do you associate with [COMPANY]?
* Partial list displayed

34
Change in Brand Attributes*
• Perceptions of ‘Customer-oriented’ and ‘Variety’ increased for all sites from
pre- to post-study
• Macy’s and Nordstrom had decreases in brand association with ‘Exclusive’
and ‘High quality’
10%
-10%
-26%
6%
4%
27%
-15%
-17%
0%
21%
19%
6%
11%
13%
-3%
23%
0%
12%
15%
25%
Macy's N=50
Nordstrom N=49
Urban Outfitters N=50
Lands End N=51
Customer-oriented
Exclusive
High quality
Value
Variety
Which of the following attributes do you associate with
[COMPANY]?
* Partial list displayed

Usability Testing
• Web & Mobile Task-based Testing
• UX & Usability Benchmarking
• Competitive Research
• Concept Testing
IA & UX Design Validation
• Navigation Optimization - Card Sorting
• Findability - Tree Testing
• Design Research - Click Testing
• Impression Testing - Timeout Testing
Customer Experience Measurement
• Voice of Customer- Live Intercept surveys
• Customer Feedback & Sat- Online Surveys
• Web Analytics Integration
35
Introduction to UserZoom
Suite of tools For
All stages of your product
life cycle

37
Summary
• Benchmarking is valuable to investment decisions
• Benchmarking can give you unparalleled insight and a
competitive edge
• Nearly every customer perspective can be measured
• Partner, Tool & Audience selection is key to success
• Retail is audiences are very measurable for online user
experience & offline in-store experience
• Task Metrics are key to conversion improvement
• Brand Metrics are very measurable via Remote Testing
• Lift in Brand Perception by a website can be HUGE
• Know if your website is presenting the desired Brand
Attributes

How to Benchmark Your Online Customer Experience Against Competition

More Related Content

What's hot (20)

Viewers also liked (7)

Similar to How to Benchmark Your Online Customer Experience Against Competition (20)

More from UserZoom (20)

How to Benchmark Your Online Customer Experience Against Competition

Editor's Notes