Usability Testing Report

NEXTRAVEL
Usability Testing Findings and Recommendations

February 2012

Usability research goals

• Primary usability research goals
• Observe Tumbler user experience to determine:
 whether evaluators understand the tumbler functionality
 how evaluators perceive and interact with the tumblers
 Interest of using tumblers to leverage their searches
• Observe Detail View user experience to determine:
 whether evaluators can easily find out the detail view
 how well the detailed information can serve their needs
• Observe Filter/NLA user experience to determine:
 whether evaluators can easily locate the filter/ NLA button
 how well the filters/NLAs can help narrow results
• Observe Map user experience to determine:
 whether evaluators can easily locate the map functionality
 whether evaluators can accurately interpret all the information in the map
 Interest and engagement with the map
• Observe Ambiguous Search user experience to determine:
 whether evaluators understand the ambiguous search functionality
 how evaluators perceive and interact with the recommendations in the tumblers
 Interest of using Ambiguous Search

©2012, Proprietary and Confidential property of Nextag, Inc.
2

Key insights: summary

• Key insights:
• Tumbler Search: clearly understood, but not always useful
• Detail View: considered useful and comprehensive
• Filter/NLA: considered useful but not easy to find out
• Map: not easy to find or understand
• Ambiguous Search: consistent with evaluators’ expectations, but
the quality was not good enough

3

Key insights: Tumbler Search

• Tumbler service clearly understood
• All evaluators could use tumblers to narrow their search results
• 7 out of 9 evaluators thought it was cool to search hotels by using tumblers
 “I think the spinners are a good idea. It does actually help you find a hotel quicker
that close to particular areas that you would like to get to.”----P2
 “I definitely like using the spinners. I've never used anything like this before. The
spinners were so convenient and easy to use, even for a first time user of the
website, like myself. ” ----P8
• Even though P2 said the tumblers were cool to use, she also mentioned to
get rid of the tumblers:
 “It's frustrating me that I cannot see this(the hotel list) in a bigger way. I don't see a
way to get rid of this(the tumblers) either. I would really get this(tumblers) a little
smaller… maybe just focus on one at a time. ”----P2
• Only P7 expressed strong dislike of using the tumblers because she
couldn’t stand too much movement
 “I wish to get rid of the screen(tumblers) and only look at the hotel list below the
spinners. I would rather have only one thing move at a time. I wish the movement
on the hotel list could be toned down.”----P7
• One out of 9 evaluators didn’t express explicitly his attitude for using the
tumblers to narrow searches
 “I don’t know how much I like the spinners.”

4


• Thoughts
 In the usability tests, evaluators were dictated to use the tumblers to
accomplish certain tasks
 The tumblers took up from 1/3 to 1/2 of an evaluator’s screen vertically
 Even though most of them felt “cool” about using the tumblers, we
don’t know how much they would like to keep using the tumblers for all
hotel searches
• Recommendations
 Need to further explore whether should make the tumblers hidable

5


• Neighborhood feature not always useful
• 3 out of 9 evaluators thought the neighborhood feature was only
useful for people who had been to the place before
 “Show neighborhood function only to people who have been to the place
before. The neighborhood function looks annoying and stupid to me.”----P4
 “Have a prompt saying 'have you been to this place before?', and use that to
control the comp up of neighborhoods and landmarks. ”----P4
 “I can only recognize Loop, Museum Campus and Airport of the neighborhoods
because I’m not living there. The neighborhood tumbler can get confusing if I’m
just casually browsing. The least useful feature for me is the neighborhoods.” ----P6
 “I’ll use the neighborhood and landmark function if I know the city.”----P10

• One evaluator(P7) skipped task 2, and she strongly disliked using
tumblers
• Even thought one evaluator(P9) claimed he thought the
neighborhood feature was useful, he didn’t use it to find a hotel
before asked explicitly to
• Recommendations
 Experiment using landmarks as the default content for the 2nd tumbler
 Shorten the neighborhood list
6


• Landmark list too long
• One evaluator said the landmark list in the 2nd tumbler was too
long and needed too many clicks
• Most evaluators thought it was helpful to have the landmark
information in a city
• Recommendations
 Shorten the landmark list to only give most famous landmarks
 Rank landmarks by its popularity

7


• Tumbler names/links not clear and easy to see
• P4 scrolled the “Destination” tumbler to look for Loop neighborhood, and
she said she didn’t notice the “Destination” tumbler was the “City”
tumbler [4:30]
• P6 said it’s hard to see the links above the tumblers
 “Oh…I see! You can change this(the 2nd tumbler) to landmarks! That’s interesting. I
didn’t know that. “
 “If they(filter options above the tumblers) were a little easier to see, I would have
started using them sooner.”

• Recommendations
 Change the name destinations to cities
 Make links above the tumblers more prominent

8


• Scrolling not fast enough
• One evaluator thought the tumblers were a little sticky, and 3
other evaluators tried to type to accelerate the scroll of the
tumblers
 “They got a little bit stuck, but they are not so bad.”----P2
 “I would like to type and see the spinners scroll to the destination.” ----P5
 “It’ll be cooler if I can just type to let the spinners spin.”----P6
 “I want the tumblers to go faster because it took a long time going from A to Z.”---
-P10
• One evaluator wished that the tumblers could keep scrolling
when he pressed the arrows
• Other evaluators didn’t see any problem of scrolling the tumblers,
and some of them thought the scrolling was very convenient and
seamless.

• Recommendations
 Enable type to accelerate the scroll

9


• Scrolling surprises
• One evaluator dragged the tumblers to scroll, but she
unconsciously clicked the empty space somewhere in the
tumbler and surprisingly found that the tumbler spinned in the
reverse direction(P2 [10:30])
 “Sometimes the spinners are a little funny. I think the spinners is fine…Maybe a little
bit tweaky” ----P2
• One evaluator used the scroll arrows in the 1st tumbler and found
it was a little frozen (P3[00:15])

• Recommendations
 Clearly inform users all the ways they can scroll or possibly move the
tumblers

10


• Defaulted sort order not clear
• 2 evaluators(P2 & P9) changed the sort order from “Best Match”
to “Price low-high”
• 2 evaluators questioned the sort order of hotels
 “It gives you hotels recommended. I’m not sure what they based that on. ” ----P5
 “to be helpful to know why you recommend it. Ur…probably values, right?”----P9
• One evaluator said she might want to see the hotel list ordered
by price
 “They(the hotel list) are sorted by best match. I guess that's OK. I may want to look by
price though. ” ----P8

• Recommendations
 Further explore what’s the best default order

11


• Poor readability
• Some evaluators reported problems of reading the contents in
the tumblers
 One evaluator said his first impression was that the website was too text-heavy(P3)
 One suggested to use another color, like yellow or light orange, for the
highlighted search result in the tumblers to differentiate it from other un-
highlighted text

• Recommendations
 ????I don’t know…it’s the problem of using tumblers

12

Key insights: Detail View

• Detail View is useful and necessary
• almost all evaluators liked the information in detail view
 8 evaluators thought the detail view gave them all the information that
they could possibly need. Only one evaluator said he disliked the detail
view and it was not clean, but he didn’t give any reasons for this
opinion.
 All evaluators could easily find
how to open the detail view,
get hotel rates and book a
hotel. 2 evaluators mentioned it
would be more desirable if
they could book a hotel
directly from the website.

13


• Detail View problems
• 2 evaluators expected to see long and detailed descriptions
 2 evaluators found there were only one or two sentences for the hotel description,
so they complained it was too short
• One evaluator wanted to know breakfast information
• 2 evaluators suggested other ways of displaying amenities
 One would like to have both amenity icons and an amenity list
 One suggested not to show the icons of unavailable amenities---” I don't see the
points of actually displaying them, if something is not available. It created
confusion. If something is not available, either put a line through it or throw it. ”(P3)

14


• One evaluator clicked the highlighted hotel name in the 3rd
tumbler to open the detail view
 “That would maybe open up a pop-up window, but it's not.”---P8

• Thoughts
 Once the tumblers become the main focus, user will expect to get all
the information from the tumblers
• Recommendations
 Need to further explore all the advantages and defects of using 2
exactly same hotel lists

15


• most evaluators thought the star ratings were guest/user ratings
 One evaluator expected to see user reviews in detail view by clicking the hotel
stars, and he was surprised that he couldn’t find any recommendations for giving
3 stars to the hotel

• Recommendations
 Include user rating contents from TripAdvisor

16

Key insights: Filters/NLAs

• Filters/ NLAs are clear & easy to use
• all evaluators liked the filters/NLAs
 One said the filter/NLA options covered all the thing he wanted
 One said the filters/NLAs were perfect and extremely clear. She rated it
ten out of ten.

17


• Filters/ NLAs problems
• 2 evaluator said the filter/NLA button might be hard to see
• Some evaluators said the amenity icons were hard to recognize
when not highlighted
 “At first it (the pool icon) looked like jacuzzi, but if you hove over, it would tell me.”-
---P2
 “That’s a pool?! Could be a little clearer about that!”----P9
 Some evaluators didn’t notice there were tool tips until they 2nd time hovered over
amenity icons
 Evaluators were guessing the meanings of each amenity icon before they
checked through all the tool tips
• Some evaluators thought star ratings were guest/user ratings
• Recommendations
 Further explore ways to place the filter/NLA button
 Re-design amenity icons to make them easier to recognize
 Include user ratings as another filter/NLA option

18


• Narrow it?
• Immediately, P3 clicked on “Narrow your results” to find a hotel
around Navy Pier, even though he had already noticed &
opened the landmark list
closest to the beach
• Thoughts
 Some well-known travel websites allow users to select hotels close to a
certain neighborhood or landmark under the filters/NLAs

19


• Narrow it?
around Navy Pier, even though he had already noticed &
opened the landmark list
closest to the beach
• Thoughts
 Some well-known travel websites allow users to select hotels close to a
certain neighborhood or landmark under the filters/NLAs

20

Key insights: Ambiguous Search

• Ambiguous Search approximated user expectations
• 7 out of 9 evaluators thought the tumblers should show places
that actually had nice ocean beaches when they searched
“beach”
 the 7 evaluators seemed very excited about the ambiguous search feature. One
said the function seemed unique.
 the 7 evaluators checked the entire region list to prove their initial guess, most of
them questioned the recommendation quality since they saw Arizona, Indiana
and Kentucky were also in the list
• Recommendations
 Tell users explicitly what will happen after searching ambiguous
keywords
 Improve the search quality

21


• 2 out of 9 evaluators couldn’t understand the defaulted
recommendations in the tumblers by searching “beach”
 One couldn’t understand why the 1st tumbler changed from a city list to a region
list after searching “beach”. He said it was very confusing and not user-friendly.
“It's putting me in Florida now, which I don't understand. It(the 1st tumbler) should
be a search filter by actual city and I don't know where it is. It's not user friendly. ”--
-P3
 One said the search was not clear.” It seemed to go to the destination with the
word ”beach” in it, not the beach I want to go to.”----P10
• Recommendations
 Inform novice users the website is capable of ambiguous searches

22


• One evaluator disliked the default recommendation by searching
beach
 “I can see why pick Miami Beach because that was… I don't know…maybe it's
the most popular search under the word beach. I don't know Miami Beach is the
destination I want to go. Maybe I want to go to West Palm Beach or somewhere
that just simply has a beach, like St. Augustine”----P5
• Thoughts
 No users knows how the search engine works in our website (our
website hasn’t established its reputation yet)
 The recommendation in the tumblers looks too strong and pushy, but it
doesn’t give any explicit reasons
 Personalized recommendations are more desirable
• Recommendations
 Need further explore ways to give “desirable recommendations”

23

Key insights: Map

• Map feature is desirable
• All evaluators thought the map feature was great
 Some evaluators liked clean maps and they thought no more information was
needed in the hotel map
 Almost all evaluators could properly interpret the information in the maps

24

Key insights: Map

• Map problems
• 6 evaluators wished the map button could be more prominent
 One said she didn’t notice the map until she was asked to find it out explicitly by
the task
 One wished the map could be more obvious because he almost didn’t see it
• one evaluator(P10) didn’t find the, and he supposed the map in
detail view was the one mentioned in the final task
• 3 evaluators had problems of closing the map
 The map close button went down below the bottom of one evaluator’s browser,
so he couldn’t find how to close the map(P6[21:35])
 One evaluator wished the map close button could be bigger
• Recommendations
 Further explore ways to place and design the map button/icon
 Make the map shorter or place the close button in the upper right
corner of the hotel map

25

Key insights: Map

• Map problems
• All evaluators didn’t notice the city map before they were asked
explicitly to tell the intention of 2 maps. 7 evaluators precisely got
the function of each map.
• Two evaluators didn’t answer all the map questions
• Recommendations
 Provide description about each map’s functionality
 Experiment using different markers in the city map and the hotel map
 Give city name pop-ups when users mouse over the markers in the city
map

26

Key insights: Map

• Map problems
• One evaluator(P9) mentioned he would like to see more hotels
instead of one or three, suddenly he noticed the map was
connected to the tumbler above it
• One evaluator(P5) complained that the information in the pop-
ups was the same with the 3rd tumbler list, and he would rather to
have no pop-ups in the map
Recommendations
Furtherexplore the
advantages and
defects of using
tumblers to navigate
the map

27

Key insights: Map

• Map problems
• Some evaluators said they would also like to see major
attractions, promotions or restaurants in the hotel map
• One evaluator said he would like to see public/private
beaches in the map when he searched “beach”
• Recommendations
 Include more information to make the map more useful

28

Appendix

APPENDIX

29

Participants and test procedure

• Web site participant attributes

• Online usability sessions:
o 9 participants: 4(female) + 5 (male)
o Ages: 4(18 – 34) + 5(35 – 55)
o Income: $40K – $100K
o Averaged Web Experience

• Usability test procedure
• Online web site research conducted via usertesting.com
o Evaluators completed several directed searches for hotels in Chicago,
then reviewed the findability and usability of Detail View and
Filters/NLAs. Evaluators also completed a directed ambiguous search
for hotels in places that had beaches, then reviewed the findability and
usability of Map.

30

Test plan
Scenario: Imagine you are looking for hotels for an upcoming vacation. You are considering Chicago because there are rich things to see and do. You are
also thinking about going to beaches to enjoy the sunshine and water. You will be using a travel website to research and book quality hotels with good price.
Please use Safari or Chrome for all the tasks.

Task 1. Go to [http://travel.nextagqa.com/] and search for a hotel in Chicago. When you’ve found a hotel you like, briefly describe what you found and why you
like it. You do not need to make a booking.
1.What is your initial impression of this site?
2.Were you able to find a hotel you like?
3.What site features, if any, were most useful?
4.What site features, if any, were least useful?

Task 2. Making sure you are still on the search results page, use the 3 Spinners below the search bar to find a hotel in the Loop neighborhood of Chicago and
another hotel close to an attraction called Navy Pier in Chicago, briefly describe what you found.
1.What information does each spinner provide?
2.What, if anything, did you like about using the spinners to find the hotels?
3.What, if anything, did you dislike?
4.How can the process of searching using the spinners be improved?

Task 3. You’d like to do more research about the hotel you found near Navy Pier:
1.Is there a way to look at detailed information about the hotel?

Task 4. If you haven’t done so already, please click on the hotel name in the yellow frame with a green border:
1.Is this the information you expected? Please explain.
2.Is there a way to look at the rates for this hotel?
3.Is there a way to book this hotel?

Task 5. Go back to the home page:
1.Is there a way to only look at 3 star hotels with pools?
2.If you found a way to only see 3 star hotels with pools, were the filter options clear?
3.What would you do to improve the process of filtering hotels?

Task 6. Imagine you’d like to find a hotel close to a beach. Type ”beach” into the search box, then use the spinners to find a hotel you find interesting. Briefly
explain what you found.
1.Is this what you expected before you started to search? Please explain.
2.What, if anything, did you like about using this search method and spinners to find the hotel?
3.What, if anything, that bothered you about finding an interesting hotel?
4.How could the process of searching using the spinners be improved?

Task 7. You’d like to stay at the hotel closest to the beach.
1.Is there a way to see the location of the hotels?
2.What do you think the two maps are for?
3.What do you think the green markers on the map mean?
4.What do you think the red markers on the map mean?
5.What other information would you like to see on the map?

31

Testing Video Links
P1 Testing Video
P2 Testing Video
P3 Testing Video
P4 Testing Video
P5 Testing Video
P6 Testing Video
P7 Testing Video
P8 Testing Video
P9 Testing Video
P10 Testing Video

©2011, Proprietary and Confidential property of Nextag, Inc. 32

Usability Testing Report

More Related Content

Viewers also liked (14)

Similar to Usability Testing Report (20)

Recently uploaded (20)

Usability Testing Report