UXG Example Report

ABC Center User Testing Report

Client: ABC Center for Research
Website: http://www.abccenter.org
Contact: John Walton
Consultants: Bazler, Jae Kim, Erik Brook, Heather Beery, Lora Ramseier
Date: January 15, 2007

Table of Contents

ABC CENTER USER TESTING REPORT........................................................................................................... 1
TABLE OF CONTENTS .......................................................................................................................................... 2
EXECUTIVE SUMMARY ....................................................................................................................................... 3
STUDY DETAILS ..................................................................................................................................................... 4
PURPOSE OF STUDY ................................................................................................................................................. 4
METHOD .................................................................................................................................................................. 4
PARTICIPANTS ......................................................................................................................................................... 4
PROCEDURES ........................................................................................................................................................... 4
TASKS ...................................................................................................................................................................... 5
RESULTS .................................................................................................................................................................. 5
TASK PERFORMANCE.......................................................................................................................................... 7
OBSERVATIONS AND RECOMMENDATIONS................................................................................................ 9
SCREEN LAYOUT/DESIGN ........................................................................................................................................ 9
TERMINOLOGY ...................................................................................................................................................... 11
CONTENT ORGANIZATION ..................................................................................................................................... 11
NAVIGATION ......................................................................................................................................................... 12
OPERATING SYSTEM AND BROWSER ISSUES ......................................................................................................... 15
POST-TEST SATISFACTION RATINGS........................................................................................................... 16
SITE SATISFACTION RATINGS ................................................................................................................................ 16
SATISFACTION QUESTIONNAIRE – POST-TEST QUESTIONS ................................................................................... 17
Question 1 – What do you consider the most valuable aspect of the system? ................................................. 17
Question 2 – What is the biggest problem with the system?............................................................................ 17
Question 3 – Additional comments .................................................................................................................. 17
APPENDICES.......................................................................................................................................................... 19
APPENDIX A – TESTING PROTOCOL SCRIPT........................................................................................................... 19
APPENDIX B – PARTICIPANT CONSENT FORM ....................................................................................................... 20
APPENDIX C – VIDEO RELEASE FORM................................................................................................................... 21
APPENDIX D – SATISFACTION QUESTIONNAIRE .................................................................................................... 22
APPENDIX E – SPACE UTILIZATION ....................................................................................................................... 23
APPENDIX F – IMAGE FONTS ................................................................................................................................. 24
APPENDIX G – CONTRAST BETWEEN TEXT AND BACKGROUND ........................................................................... 25
APPENDIX H – TERMINOLOGY ............................................................................................................................... 26
APPENDIX I – LINK FORMAT ................................................................................................................................. 27
APPENDIX J – MOUSE-OVER EFFECT..................................................................................................................... 28
APPENDIX K – LINKING TO PREVIOUS PAGE ......................................................................................................... 29
APPENDIX L – OPERATING SYSTEM AND BROWSER ISSUES .................................................................................. 30
APPENDIX M – BROWSER COMPATIBILITY ISSUES ................................................................................................ 31

User Experience Group Indiana University 2 of 31

Executive Summary

In March 2005, a usability study of the ABC Center website was conducted with fifteen
participants in the Bloomington, Indiana area. Results showed that users were generally
satisfied with the amount and quality of the content available on the ABC site. However, a
number of usability issues were identified that hindered locating information, and the site was
perceived as being unprofessional in appearance.

One of the greatest concerns for usability in the ABC site was the lack of a consistent, reliable
navigation system. Users were often left without navigation options to return to the previous
page or to the main menu, and were forced to use the browser buttons to navigate the site. This
issue was compounded by the fact that some links took the users unexpectedly out of the site.

The overall unprofessional appearance of the site was another concern. When searching for
scientific research, the perceived credibility of a site will play a large role in whether people are
willing to use the site and trust the information provided. Participants expected the images
used in the site to be of higher quality, page space to be utilized more efficiently, and the
content to be formatted correctly (capitalization, text alignment, etc.). Even though these issues
may seems somewhat minor, the combined effect reduces the sites credibility in the users’ eyes.
Other usability issues included linking to software-specific files without warning and
unpredictable content presentation format.

This report describes the purpose and methods of the study, and provides a summary of the
observations made from each testing session. Recommendations are made for the ABC Center
web development team to consider as they prepare for the redesign of the website.


Study Details
Purpose of Study
The ABC Center website is a resource for students to find research studies on the web. The site
provides a variety of features including a research article database, links to other online
research sources, and a directory of researchers. A new site design is going to be introduced in
July, 2005, and the ABC Center web development team wants to explore opportunities to
improve the website by obtaining user feedback. As part of the usability evaluation for the
ABC Center, user testing was conducted in March of 2005, in Bloomington, Indiana.

Method
Usability testing methodology1 was used to evaluate the website. This approach involved
having authentic users perform authentic tasks using the system, while evaluators observed and
recorded their actions and comments. Users were asked to perform a think-aloud protocol2 to
help the evaluators understand their behaviors and gain insight into the design of the website.
Sessions were performed on an individual basis with each session lasting approximately 1 to 1
½ hours. ABC Center provided users with $50 gift certificates in return for their participation.
Following the last session, qualitative and quantitative data were analyzed and
recommendations for redesign were made.

Participants
Participants recruited for the testing sessions were identified and recruited by the UXG team.
An effort was made to target participants who fit a wide range of user profiles due to the
heterogeneity of the site’s potential users. The goal was to recruit a sample of 16 participants
to achieve adequate representation across the following four demographic categories:

1. Gender – approximate equal split between male and female
2. Age – three age ranges (20-39, 40-59, 60+)
3. Computer experience – novice (little to no computer use), intermediate (office
applications, web), or expert (complex applications and/or programming)
4. Experience with research websites

A total of 15 participants were tested including 7 men and 8 women. Five participants were in
the 20-39 year age-range, six were in the range of 40-59 years of age, and four participants
were over 60.

Procedures
Participants were read an introductory protocol at the beginning of the testing session (see
Appendix A for script) and asked to read and sign two consent forms. The first consent form
(see Appendix B) is a standard consent form indicating the participants’ understanding of what
will be expected of them during the sessions and their willingness to participate. The second
consent form (see Appendix C) is a video release form indicating the participants’ awareness
that the sessions are videotaped and granting their permission for the video to be used in the

1
Dumas, J. S., & Redish, J. C. (1993). A practical guide to usability testing. Norwood, NJ: Ablex.
2
Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data (Revised ed.). Cambridge,
MA: MIT Press.


evaluation and reporting of results. Participants were given a copy of the consent forms for
their records.

Each participant completed 10 tasks using the abccenter.org website. Tasks were completed
one at a time and were recorded as either 1) success with ease, 2) success with difficulty, or 3)
failure to complete the task. Success with ease included those tasks that were completed on the
first or second try. Success with difficulty included tasks that participants adequately
completed but more effort and persistence was needed. For example, trying numerous paths to
locate necessary information before finding the correct screen. Failure to complete a task was
given if the participant gave up on the task or failed to locate the correct information. In some
“failure” cases, participants may have been satisfied that they completed the task; however, the
correct information was not located or portions of the task were not completed.

Following the session, the users were asked to rate their satisfaction with the system using the
Survey of User Satisfaction developed by Digital Equipment Corporation (see Appendix D).
Tasks were performed using Internet Explorer 6.0 and the Windows XP operating system on a
Dell, Pentium 4, 3.80 GHz computer.

Tasks
The ABC Center team developed a draft list of 10 tasks for testing based on key site
information and features. UXG consultants reviewed and refined the tasks in order to create
appropriate scenarios for user testing. Tasks were presented in random order to eliminate order
effects from influencing the testing results.

Results
The results of the study have been divided into four sections:
1. Task performance (e.g. successes and failures)
Tasks are listed and the number of participants experiencing successes and failures are
noted.
2. Observations and recommendations
The results of the study are presented in a table format. The table includes three
columns of information: Observations, Interpretations, and Recommendations. Each
column is described in more detail below:
• Observations – an objective description of participants’ actions and comments
during a session.
• Interpretations – a proposed explanation for participants’ observed behaviors and
justification for recommendations based on known design principles and rules.
• Recommendations – suggestions for maintaining aspects of the current design that
work, changing aspects that are problematic, and considering alternative
possibilities for presenting information to the user.
The data within this table are organized into categories that represent the major types of
issues raised during the study. The categories include:


• Screen Layout/Design – Items concerning the placement of elements on the screen,
use of screen space and opportunities for modifying items to increase user
understanding of presented information.
• Terminology – Items concerning the user’s understanding of the language used on
site.
• Content Organization – Items concerning the organization of information within the
site.
• Navigation – Items concerning the users’ ability to maneuver throughout the site.
• Operating System and Browser Issues – Items concerning compatibility and
consistency of site presentation across major computer configurations.

3. Post-test satisfaction ratings and comments
The results of the SUS (System Usability Scale) questionnaire and a brief description of
how the overall SUS scores are calculated is presented. Participants’ comments are
included.


Task Performance
Task Success Success with Fail
with Ease Difficulty
1 You are looking for employment at the ABC Center. Find out if 12 2 1
there is anything open at this time that you could apply for. Are
you interested in the position?
2 You are interested in taking an online course offered by the ABC 11 2 2
Center. Find a course in “Referencing in APA Style.”
3 You are writing a research paper, and you realize that you are 7 3 5
missing the author information for one of your sources. Find the
author of the research article “Intercultural Discourse.”
4 You have been asked to contact a researcher by your 4 7 4
teacher/professor. Find the contact information for “Ellen Miller.”
5 You are interested in visiting the ABC Center in person. Where is 13 2 0
the ABC Center located?
6 With so many online research databases to choose from, you 4 6 5
wonder how they compare. Find the discussion forum where the
users rate them.
7 After spending some time browsing through the ABC Center 2 6 7
research database, you found some articles that you might need in
the future. After logging in, find and save the research article
“Benningham DNA Study” in your “Online Bookmarks” for later
retrieval.
8 You have heard that ABC Center offers several workshops 7 5 3
throughout the year. What are the titles of the two that are offered
in May (2005)?
9 ABC Center offers several free research aid software for 12 1 2
download. Find and download “Ace Reference Tracker 2”.
10 Thank you for visiting the ABC Center website. Can you please 10 3 2
use the online form to rate us?


The graph above illustrates the percentage of users who accomplished the tasks with ease
(easy), with difficulty (hard), or failed. The tasks that caused users the most difficulty were
locating the contact information for researchers (task #4) and using the Online Bookmarks
feature (task #7).


Observations Interpretations Recommendations

Observations and Recommendations
Screen Layout/Design
The copyright information on Using the same colored text and 1. Use black or other dark
the homepage is invisible to the background does not provide any colored text for this
users since it is white text on contrast for the text to be visible, information.
white background. unless they are drag-selected with
the mouse by accident.
At least three different font and When the same information is 2. Standardize the font and
color schemes are used in the presented in different formats, it color format if the
site to graphically represent the causes inefficiency in user organization does not have
organization’s name. behavior because they have to re- a graphical logo. If it does,
interpret the data. Furthermore, a replace the current titles
chance to reinforce the with the logo.
organization’s image/identity is
lost.
The organization’s name is The organization’s name is not 3. Change the layout of the
difficult to read. easy for users read. It looks like a organization’s name.
list including; “the center,
education, research, retailing”.
The information currently The homepage does not help 4. Remove the current home
displayed on the homepage is users understand what the site is page and make the “ABC
not helpful to users. For about or where they can navigate. Center Menu” page the
example, the organization’s site’s home page.
name is shown twice, the
picture displayed is not
associated with the center, and
there is no site menu or
navigation on this page.
The “Research Time Line” link Content should be presented in a 5. Change the font of the link
is presented in a different font consistent graphical format, to be consistent with the
than the other links in the same especially if the items belong to the other links in the list.
list. same list. Any deviation would
suggest that the particular link is
different from the others in content,
importance, etc.
Too much screen space (80+ The top of a web page is valuable 6. Move the page title image
pixels) is wasted on the top of and draws the attention of the up to reduce the white
the menu page (see Appendix users. The current use of the page space.
E). layout does not utilize this
potential.
Many images used in the web Low quality images degrade the 7. Replace the existing
site are of low quality and/or overall look of the web site, making images with higher quality
stretched from smaller samples, the site and its affiliated images.
making the overall look of the organization less credible. 8. Limit the use of images
site unprofessional. Images should enhance the look of because they increase site
a site without being distracting or load time and can distract
overwhelming. users from more important
content on the site.
The participants commented Low quality text images and/or 9. Replace all text images
that some items were blurred adjusting image size (shrinking with normal text, unless
and difficult to read (see and stretching) can cause images specific design
Appendix F). to appear blurry. considerations require it
(such as within a logo).


There is no consistent format Users have difficulty knowing what 10. Choose a format for page
for content text, links, and constitutes a link, a header, plain layout, title, content text,
visited links. text and imbedded links. Without a links and visited links and
consistent format, users must apply the format
explore each item to determine if it consistently across the
is a link or not. This requires more site.
time and energy than many users
are willing to spend.
Every page begins with a title Providing a clear title on each 11. Choose a font, color, size
that describes the contents of page helps users know where they and position for the title
the page, but the format is not are and feel more comfortable and use the same format
consistent across the site. navigating the site. However, an consistently throughout the
inconsistent format can cause site.
confusion.
The repetition of frame at the The site identification is important. 12. Apply a consistent format
left hand side could be site The repetition of screen layout, to the page layout as well
identification; however, there is font size and color contribute to as the contents.
no design scheme or stylistic site identification.
treatment for the content frame.
The link on the left frame turned The purple text on a red 13. Choose a color schemes
to purple when user clicked on background does not provide that provide good contrast
the link and used the “Back” sufficient contrast and is therefore between the text and
button on the browser to go difficult to read. background.
back to the page (see
Appendix G). Participants were
not able to read the link.
Links to external sites are The main purpose of navigating 14. Replace the links with the
included in what should be the the site is to locate information site navigation structure.
site’s main navigation. about the organization instead of 15. Put the external links
linking to other sites. Most- under the “resources”
frequently used links and the site category or put them in the
navigation structure should be footer.
available on every page.
An empty check box in front of Inappropriate use of symbols or 16. Remove the empty check
the “Return to ABC Center images causes confusion. Users box and the check box
Menu” link causes confusion. may not know whether to click on animation from the links.
Participants were unsure the underlined text or to click the
whether it was a bullet point or box.
something requiring an action. The check box animation is not
On some pages, there is a relevant to the link or its content
checkbox animation in front of and is distracting to the user.
the link.
The navigation image map Using large images for navigation 17. Do not use large images
used on “About Our Center” is not an efficient way of organizing unless absolutely
takes too much screen space content on a web page. necessary. Replace the
(550 x 413 pixels). image map with a bulleted
list of links.
There are no visual cues for User could easily miss/ignore a 18. Avoid using image maps.
users to know that the 4 red link if it does not look like a link. 19. If an image map is
triangles on the corners of the necessary, provide visual
image map on “About Our cues (underline the text on
Center” page have imbedded graphics, for example) to
links. tell the user that there is an


imbedded link.
Participants clicked on some There is no visible distinction 20. Choose a font, color and
texts and then realized that it between text and embedded links. style for imbedded links
was not a link. Also, there is no consistent format and use the format
for imbedded links. consistently across the
site.
Bulleted links do not line up on Proper item alignment is an 21. Make sure all items are
the ‘Programs for Students’ important element for professional properly aligned.
page. presentation.
Most, but not all, link text is A consistent capitalization scheme 22. Change the link text to
capitalized on the ‘Programs for is an important element for the adhere to a consistent
Students’ page. professional appearance of the capitalization guideline.
site.
Terminology
Only the homepage has a Windows should be given a clear 23. Provide an appropriate title
window title and it did not identifiable title that will display in for every web screen.
describe the website clearly. the browser address bar. This will
also help the user bookmark the
page appropriately.
The “Welcome to the ABC The sentence is more like a 24. Phrase all the links
Center” link on the homepage is greeting instead of a link. User appropriately. The link text
vague as it did not tell user would not be able to tell that it is a should clearly identify the
anything about the purpose of link if the link was not underlined. purpose of the link.
the link.
Participants expected to get The manner in which the images 25. Change the title to be
access to an application by were displayed led the user to “Screenshots of Software
clicking on images under believe they could access the Applications” to tell users
“Software Applications” (see various applications shown. that those images are just
Appendix H) some screenshots from the
applications.
26. A brief description for each
image helps users
understand the content.
The names of the links to the Using a consistent word or phrase 27. Make sure the name of a
menu page are not consistent. to link to any given page will help link is identical on every
On some pages, it is called the user learn the structure of your page.
“ABC Center Welcome Page”; site and allow them to accurately
on other pages, it is called select the appropriate link no
“ABC Center Menu”. matter what screen they are on.
The menu item and the page Users are not sure where they are 28. Make sure the link reflects
title are not the same. when they click on a link and get a the title of the target page.
page with a title different from the Page titles may be longer
link they clicked. and more descriptive than
links, but key words should
be the same.
Content Organization
Contact information is buried Users should not need to browse 29. Consider placing basic
too deep into the web site and deep into the site in order to find contact information
only available after following a basic contact information. Many (address, e-mail, and
narrow navigation route. users will look for this information phone number) at the
and become frustrated if it is not bottom of each page on
easily located. the web site, or providing a


top-level link to the contact
information.
A number of links open an Users can easily become confused 30. Only contents of the
external web page within the about which site they are visiting if website should display in
content frame. an external site opens within the the content frame. An
content frame. external web page should
open in a new window so
users know that it is
another site.
A number of links on the site Users like to be informed before 31. Provide the content in a
open a PDF file, a word downloading/opening a file. They format that users can view
document, a PowerPoint file or become frustrated when they click online. That gives user the
even a streaming video without on a link and get unexpected control to choose to read
alerting the users beforehand. results. the document online or
download it.
32. Clearly label any links that
will start a download or
attempt to open an
application (such as
Acrobat Reader, Word,
PowerPoint, etc.) so users
know what to expect
before clicking.
‘Customer Interface Technology Users would like to be informed 33. Warn the users that the
Lab’ opens a Real Media file, before trying to open a file that link will open a Real Media
which requires a media player requires an application they may file, and provide a link to a
that has to be downloaded. not have. download page where the
required media player is
available.
Navigation
Participants spent too much Users expect to see a clear 34. Remove the current
time scanning the homepage navigation scheme that can help homepage from the site,
with the mouse to see what determine the next page to visit. and establish the main
links were available. Time and effort is wasted when the menu page as the
users have to explore the page to homepage.
find what navigation options are 35. If the current homepage
available. has to be kept, make sure
that the users know that
there is only one
navigation option available
by moving the “welcome”
link to the center of the
page.
A clear association between the The menu page specifically 36. Provide the bulleted list
top-level navigation presents a bulleted list of content presented in the
descriptions and sub-level information that is available in menu page for the sub-
content was not apparent to each of the top sections. This level pages.
participants. leads users to believe that these
categories will be clearly marked in
their respective sub-pages.
The current layout, however, does
not distinguish these categories.
A quick, obvious link back to The current navigation scheme lets 37. Remove the current
the menu page is not provided the users go back to the homepage from the site,


throughout the site. homepage from most of the site and establish the main
pages. However, there should be menu page as the
little need for this, since the homepage.
homepage only provides a link to 38. If the current homepage
the menu page. It is more logical has to be kept, replace the
that the users will want to return to universal link to the
the menu page from their location homepage with link to the
in order to restart their interaction menu page.
with the site.
Participants do not know where The current site design does not 39. Provide information about
they are and where they came indicate where the user is within current location to facilitate
from when navigating the site. the site. Show the user where navigation.
they are by using breadcrumbs at
the top of screen or by highlighting
the user’s location within a
persistent navigation structure.
User would feel more confident if
they know exactly where they are
and where they came from when
navigating a site.
Participants relied on the back Users frequently miss the link 40. Provide a sound navigation
button on the browser to because they would have to scroll mechanism.
navigate the site. A ‘go back to to the bottom of the page to find it. 41. Make the go back to
the previous page’ link is In addition, the link is not always previous page link
provided at the bottom of some available on every page. Users available at the same
pages, but not all. would be frustrated if they scroll position on every page.
down to the bottom of a page but
cannot get the link they expected.
Some participants could not Users normally click on an item’s 42. Make the name/title of an
locate information on the name to get more details about the item a link instead of
center’s director as they did not item instead of clicking on the imbedding the link in a
know that the link was bullet. bullet point.
imbedded in the bullet point A dark red border around the red 43. Links should be presented
(see Appendix I). rectangular bullet point is a hint to in an easy to recognize
tell users that it is a link, but it is and consistent manner.
not easy to tell as the colors are
quite similar.
Participants did not realize that The use of fancy technique is not 44. Remove the mouse-over
the image above the facility lists always appropriate as users might effect and allow users to
would change when the mouse not know how to use it. They may get more details of a
moved over an item of the list. easily miss the information if they facility item by clicking on
Participants clicked on an item do not know how to get access. it.
as the cursor turned to a finger User would be frustrated as the
when moving over the item. cursor turns to be a finger when
moving over an item, but those are
not links and they cannot get more
information by clicking on it.
On the research on retailing User would not be able to know 45. Use the title of the study as
page, links of two study reports even the title of a report unless a link and clearly labeled
have the same title. clicking on the link. the link to tell user what to
The image above the sentence It causes confusion when the site expect when clicking on
of “Click for a PDF copy of the cues the users for action that does the link.
study” is not a link although it not lead to the expected result. 46. Change the image to act
indicated to be. as a link.


Participants were confused Users expect different links to go 47. Either combine the links to
when the “shopping to different pages. Many users will be “shopping simulations &
simulations” link and the not understand that the links point software applications” or
“software applications” link took to different sections on the same separate the contents into
them to the same page. page. two separate pages.
When the user mouses over a The list format and position of the 48. Remove the mouse-over
category in the menu, it popped pop-up information create effect and use available
up a brief description of the confusion because it looks like a screen space to display
category. Participants regarded pop-up submenu. Users become the brief description right
it as a pop-up submenu and frustrated trying to click on this below each category.
wanted to click on it but the image that disappears every time 49. Change the format to
pop-up description disappeared they move their mouse. make it more like a brief
once the cursor moved away description of a category in
from the category. Participants stead of a list
moved their cursors back and
forth between the menu and the
pop-up information and then
realized that it was not a
submenu (see Appendix J).
The link at the bottom of the Bad navigation mechanisms force 50. Simplify the link at the
page always returns the user to the user to recall where they came bottom of the page to be
the previous page (see from. “back to previous page”.
Appendix K). However, Users have more processing 51. Provide a mechanism that
participants were confused by capacity for the contents when helps users to recognize
links that contained the title there is less cognitive load for where they are in the
because they did not always locating their position. website instead of recalling
recognize the names of the their path.
previous pages they visited.
‘KPMG’ link in the ‘Sponsoring Dead links damage the credibility 52. Fix the dead link to lead to
Partners’ page is a dead link. of the site and frustrates the users the correct, working site or
who wish to visit the linked page. if the site no longer exists,
remove the link and
associated text.
The ‘Marshall Field’s’ and Link names should always 53. Fix the ‘Marshall Field’s’
‘Mervyn’s’ links on the correctly refer to the site they link and ‘Mervyns’s’ links to
‘Sponsoring Partners’ page do to. Unexpected behaviors, such as lead to their respective
not take the user to the correct the wrong site opening, lead to sites. Check all links
sites. Rather, both of these user confusion and frustration. (internal and external) for
links lead to target.com. accuracy and periodically
verify that external links
still point to published
sites.
On the ‘Activities and Users expect similar behaviors 54. Group links together that
Programs’ page, various links in from links that are grouped behave in the same
the same bulleted list behave together. Furthermore, users’ manner. Provide a
differently. Some open up new mental model of the site navigation distinction or visual
browser windows, others open scheme is weakened when separation for links that
PowerPoint-like slides, and unpredictable links are present. behave differently.
others go to pages deeper in
the hierarchy.
Clicking any items on the The users expect the side 55. Fix the links so that the
“Timeline” page takes the users navigation bar to be present at all pages are displayed within
out of the site navigation frame. times. the content frame in the
Clicking on the “Home” link on browser window, leaving


the “Timeline” page takes the the side navigation intact.
users out of the site navigation
frame.
The “Home” link at the upper- Different links with the same name 56. Rename the link to avoid
left hand corner of the causes confusion. confusion with the link to
“Timeline” page is confusing. the center’s homepage.
Participants did not know
whether they would be taken to
the site’s homepage or the
center timeline’s homepage.
Operating System and Browser Issues
The red rectangle with the Some operating systems and 57. Test the most popular
imbedded link on the “About browsers are more flexible than browsers and operating
Our Center” page has a 1 pixel other in rendering code systems to make sure the
bolder to indicate that a link. site works properly.
The border does not display
properly on a Macintosh (see
Appendix I)
The checkbox image in front of Some operation systems and 58. Test the most popular
the “Return to Research Center browsers are more flexible than browsers and operating
Menu” works on PC, but the other in rendering code systems to make sure the
image did not display properly site works properly.
on a Macintosh (see Appendix
L).
The mouse-over link color does Some operation systems and 59. To increase the likelihood
not change for Netscape 7.2 on browsers are more flexible than that pages will display and
PC, Netscape 7.01 on other in rendering code function properly, make
Macintosh and Firefox 1.0 on sure that code is formed
both PC and Macintosh correctly with proper tags
computers. (always use closing tags),
etc.
60. Test the most popular
browsers and operating
systems to make sure the
site works properly. If all
browsers and operating
systems will not be
supported, provide text
stating which browsers the
site supports.


Post-test Satisfaction Ratings
Site Satisfaction Ratings
Satisfaction User Survey 1 5 Median Mean
strongly strongly
(Raw score by SUS question)
disagree agree
I think that I would like to use this 5 5 2 4 4 3 3 4 3 5 4 2 3 2 3
system frequently. 3 3.47
I found the system more complex than 1 2 5 3 3 5 1 2 3 1 3 5 3 5 1
necessary 3 2.87
I thought the system was easy to use. 4 5 2 2 3 3 4 5 3 5 3 2 2 2 4 3 3.27
I think that I would need the support of 1 1 1 3 1 1 1 2 2 1 2 1 1 1 1
an experienced person to be able to use
this system. 1 1.33
I found the system visually appealing 5 5 3 4 2 3 4 4 5 5 3 5 3 3 4 4 3.87
I thought there was too much 1 1 3 1 3 2 1 1 3 1 3 3 4 3 1
inconsistency in this system. 2 2.07
I would imagine that most people 5 5 1 4 2 2 5 4 3 5 3 2 3 1 5
would learn to use this system very
quickly. 3 3.33
I found the system very cumbersome to 1 1 4 2 2 2 1 4 4 1 3 5 3 4 1
use. 2 2.53
I felt very confident using the system. 5 4 5 3 2 4 5 5 2 5 3 3 4 5 5 4 4.00
I needed to learn a lot of things before I 1 1 1 3 3 1 1 1 3 1 4 2 1 1 1
could get going with this system. 1 1.67
Satisfaction User Survey (total SUS Score (0-100)
adjusted SUS scores by participant)
Participant 1 97.5
Participant 2 95
Participant 3 47.5
Participant 4 62.5
Participant 5 52.5
Participant 6 60
Participant 7 90
Participant 8 80
Participant 9 52.5
Participant 10 100
Participant 11 52.5
Participant 12 45
Participant 13 57.5
Participant 14 47.5
Participant 15 90
Median 60
Mean 68.67

Raw SUS scores are adjusted as follows:
* Questions 1, 3, 5, 7, and 9: adjusted score = raw score – 1
* Questions 2, 4, 6, 8, and 10: adjusted score = 5 – raw score

Calculating an individual’s total SUS score for an application:
* The sum of the adjusted scores is multiplied by 2.5 to yield an overall score in the range of 0
– 100.
Higher total SUS scores indicate greater overall satisfaction with the website.

The median and mean scores of 60 and 68.67 respectively for the ABC Center website
represent a somewhat lower level of user satisfaction than desired. Correcting the various
usability issues identified in the preceding sections of this report are likely to improve user
satisfaction with the site.

Satisfaction Questionnaire – Post-Test Questions
The table below shows participants responses to the three SUS post-test questions.
Participants wrote their responses on the SUS form and they are included here verbatim.

Question 1 – What do you consider the most valuable aspect of the system?
P1 Easy to use, straight forward.
P2 Access to rich, original research.
P3 Easy to find research based on topic of interest.
P4 Getting information.
P5 There is a lot of information.
P6 Fairly easy to find what I needed.
P7 Wealth of information.
P8 Links to cross references.
P9 Rich information.
P10 Providing easy links for additional research material.
P11
P12 This site has many research materials that are not available anywhere else.
P13 I liked the links to other research resource sites.
P14 Contact information for the research authors is a good resource.
P15 Easy to find the research I am looking for.
Question 2 – What is the biggest problem with the system?
P1 Some of the links took me out of the site completely without warning.
P2 The images that are used are not that good in quality. It makes the site look unprofessional.
P3 Organization is not clear.
P4 Make it user friendly.
P5 There seem to be a lot of space wasted – I would rather have more content than fancy titles and
graphics.
P6 Did not like the images used.
P7 It was difficult to navigate the site.
P8 The pages in the site looked too different from each other. They don’t look like they belong to the same
site.
P9 Difficult to navigate.
P10 Lack of professionalism in site presentation.
P11 N/A
P12 Full screen is not utilized.
P13 Site looked unprofessional.
P14 Navigation options are sometimes unavailable.
P15 Many linked files required programs that I did not have on the computer.
Question 3 – Additional comments
P1 Nice site, lots of information.
P2 Change the images to better ones, and the site will look much nicer.
P3
P4 About time! [do not know what the participant was referring to ]

P5
P6 It was frustrating because navigation options would disappear on me at times. Those links should be
there at all times. Also, there were instances where the links that I clicked took me to an external link. I
think those should open up new windows.
P7
P8
P9 Needs more organization.
P10
P11 I look forward to using it in the future, nice content.
P12 Overall it was fine, but I think the images should be changed to better ones.
P13
P14 Make the navigation bars available at all times. This will make the site much easier to use. I liked the
content available on the site, lots of original research.
P15


Appendices
Appendix A – Testing Protocol Script
Participating in Usability Sessions
Thank you for agreeing to participate in the usability study of the ABC Center website.
Before we begin, I will briefly go over what a usability session consists of, what will be
expected of you, and what our goals are in conducting this study.
The first thing that I want to make clear is that we are testing the application, NOT you. If
you find errors or if you have any difficulties with the application, it is very likely that other
people visiting the site will also experience those same difficulties. We will use this
information to better understand how we might improve the site.
During the Session:
1. Tasks:
You will be asked to perform a series of tasks using the ABC Center website. The
tasks are structured to determine if the most important information and features of the
site are easy to locate and use. We ask that you try to accomplish the tasks without
assistance, as if you were at home trying to find the information on your own. This
gives us a better idea of the things that work well or the difficulties people experience.
Anytime during the session, if you find yourself thinking that you would quit the task,
use help, contact someone for assistance, or visit another site to accomplish your goal,
please let us know.
2. Test Facilitator and Observers:
One member of the User Experience Group will act as the test facilitator assisting
you in getting started and answering any questions you may have. Another member
of the User Experience Group will observe and take notes. Representatives from the
ABC Center team may also be present to observe the session and take notes.
3. Think Aloud Protocol:
To help the observers understand the way you use the application, you will be asked
to “think aloud” as you complete the tasks. This simply means that we ask you to
talk about what you are doing, what you are looking for, clicking on, wishing you
could find, etc. In other words, any task-related thought that comes into your mind
we would like for you to share aloud. Some people find this easy right away, while
others need to be reminded a little. It may seem a bit odd to talk about every step that
you are doing, but it really helps the observers understand how you are interacting
with the application and how the application is working.
After the Session:
1. User Satisfaction Questionnaire
The user satisfaction questionnaire is a short questionnaire that asks you about a few
general aspects of the system. It will only take a few minutes for you to complete.
2. Questions
If you have any further questions for the facilitator or members of the ABC Center
team, you will have the opportunity to ask them at that time.


Appendix B – Participant Consent Form

Participant Consent Form
The purpose of this usability study is to evaluate the design of the ABC Center
website. We are interested in determining if people can accomplish common
tasks and easily find information using this website. The session will not ‘test’
you or your ability, rather the session will test the website in order to provide
information on areas that might be improved. Please be advised that there are
no foreseeable risks associated with participation in this session.

During this session, you will be asked to complete some tasks using the ABC
Center website and fill out a user satisfaction questionnaire. As you complete
the tasks, members of the User Experience Group and ABC Center will observe
and take notes. In addition, the session will be captured on video for future
review. The session will last no longer than one hour and fifteen minutes.

If for any reason you are uncomfortable during the session and do not want to
complete a task, you may say so and we will move on to the next task. In
addition, if you do not want to continue, you may end the session and leave at
any time.

Approximately 16 people will participate in this study. Results from all sessions
will be included in a usability report to be presented to ABC Center. Your name
will not be included in the report nor will your name be associated with any
session data collected.

If you wish to speak with someone about your participation in this study, or if
you feel you were not treated as described above, please contact the User
Experience Group manager at 812-855-4499.

I, ______________________________________________, have read and fully
understand the extent of the study and any risks involved. All of my questions,
if any, have been answered to my satisfaction. My signature below
acknowledges my understanding of the information provided in this form and
indicates my willingness to participate in this user testing session. I have been
given a blank copy of this consent form for my records.

Signature:______________________________
Date:________________


Appendix C – Video Release Form

Video Release Form
The signature below indicates my permission for University Information
Technology Services User Experience Group of Indiana University to use video
footage recorded during the usability session conducted for

ABC Center Website on _______________, 2005

in which I served as a participant.

My name will not be reported in association with session results nor will my
name be included on the video footage. This video footage may be used for the
following purposes:

• Analysis of research and reporting of results
• Conference presentations
• Educational presentations
• Informational presentations

I will be consulted about the use of the video recording for any purpose other
than those listed above.

There is no time-limit on the validity of this release nor is there any geographic
specification of where these materials may be distributed.

This release applies to video footage collected as part of the usability session
listed on this document only.

I have been given a blank copy of this release form for my records.

Name (please print): Date: / /

Signature:

Address:

Phone: E-mail:


Appendix D – Satisfaction Questionnaire
System Usability Scale
© Digital Equipment Corporation, 1986. w/ Revision by Usability Consulting Services, 2002.
Strongly Strongly
disagree agree

1. I think that I would like to
use this system frequently

2. I found the system more
complex than necessary

3. I thought the system was easy
to use

4. I think that I would need the
support of an experienced person to
be able to use this system

5. I found the system visually appealing

6. I thought there was too much
inconsistency in this system

7. I would imagine that most people
would learn to use this system
very quickly

8. I found the system very
confusing to navigate

9. I felt very confident using the
system

10. I needed to learn a lot of
things before I could get going
with this system


Appendix E – Space Utilization

Too much screen space (80+ pixels) is wasted on the top of the menu page.

Move the location of the page title image up to reduce the white space that occupies the top of
the page.


Appendix F – Image Fonts

The participants commented that some items were blurred and difficult to read. Low quality
text images and/or adjusting image size (shrinking and stretching) can cause images to appear
blurry.

Replace all text images with normal text, unless specific design considerations require it.


Appendix G – Contrast Between Text and Background

Figure.1 Figure.2

Figure 1 shows what a link looks like on the red frame. The link turned to purple when user
clicked on the link and used the back button on the browser to go back to the page (Figure 2).
Purple font on a red background does not provide sufficient contrast and many participants
will have difficulty reading the text. Choose a color scheme that provides good contrast
between the background and text, links, and visited links.


Appendix H – Terminology

Figure.1 Screenshot of the Virtual Showcase page

Figure.2 A screenshot of an application

When clicking on an image under the “Software Applications” session (see Figure 1
above), users expect to get access to the application. Users tried to click on the image
(Figure 2) to navigate the system and then realized that it was just a screenshot of the
application.


Appendix I – Link Format

The image above shows the current design scheme on the ‘About Our Center’ page. Users
easily miss the links since bullet points are not a common place to locate links. The dark red
border may have been meant to indicate an embedded link, but the border is difficult to see
due to low contrast between the colors.

The image above shows what the rectangular bullet points look like on Macintosh computers
using Internet explore 5.2 or Safari 1.2.4
It is difficult for users to get more details since there is no visual cue (border around the bullet
point) to tell user that the bullet points are links and using bullets as links is not a common
practice. If users notice that links are available, it will likely be due to accidentally mousing
over the bullet point and noticing the cursor change.

The example above uses the name/title of an item as a link instead of the bullet point. Using
the name/title to link to more details of an item is a common method that users are familiar
with, and so they would be less likely to miss the link. In addition, applying a consistent
format (such as underlining) to all links will help users easily distinguish links from other
items on the site.


Appendix J – Mouse-Over Effect

Figure.1

Figure.2

Figure 1 above shows the appearance of the center menu page. When the user moves the
mouse over a category of the menu, the category was highlighted and it popped up a
short description about the category (see Figure 2). The short description displayed in
list format; users regarded it as a pop-up submenu and tried to click on it. They got
frustrated as the “submenu” disappeared when the mouse moved away from the
category.


Appendix K – Linking to Previous Page

The link at the bottom of the page always returns to the previous page. However, participants
were confused by links that contained the title of the target pages (as shown in the example
above) as they did not always recognize the names of the previous pages they visited.

Simplify the link at the bottom of the page to “back to previous page”.

In addition, provide a mechanism, such as the breadcrumbs shown above, that helps users to
recognize where they are in a website instead of recalling their path.


Appendix L – Operating System and Browser Issues

There is a checkbox image in front of the Return to Research Center Menu link on
Windows using Internet Explore 6.02, Firefox 1.0.2, and Netscape.

The checkbox image does not display correctly on Macintosh computers using
Internet Explorer 5.2

The checkbox did not show up on the Macintosh computer using Safari 1.2.4


Appendix M – Browser Compatibility Issues

Using the most recent version of Internet Explorer on both the Mac and pc, the links change color with mouse-
over to indicate that they are selected and the cursor is changed to indicate a hotspot.

In Netscape 7.2 (pc), Netscape 7.01 (Mac), or Firefox 1.0 (pc and Mac), however, the mouse-over does not
work, and the cursor stays the same.


UXG Example Report

More Related Content

Similar to UXG Example Report (20)

More from Sean Connolly (9)

Recently uploaded (20)

UXG Example Report