Usability Report - Discovery Tools
5 likes1,891 views
The usability testing report evaluates the discovery tools Summons and Encore for the University System of Maryland, revealing that Summons proved to be more user-friendly than Encore, with participants finding Encore more confusing and frustrating. Recommendations include improving user feedback and reorganizing screen elements for better clarity based on identified usability issues such as inadequate search hints and poor visibility of important links. Overall, the report suggests significant improvements are needed for both tools to provide a better user experience.
Usability Report - Discovery Tools
- 1. USABILITY TESTING REPORT: DISCOVERY (SEARCH) TOOLS OF ENCORE AND SUMMONS Authors: Heather Mathieson, Debbie Shultz, Nikki Kerber IDIA 642 - Lucy Holman May 12, 2010
- 2. Table of Contents Executive Summary Participants Methodology Results Recommendations Appendices Heat-map Wireframes Consent Form Faculty Screener Graduate Student Screener Script Task List
- 3. Executive Summary This report contains the results of a user research project conducted between April 14 and May 12, 2011, to develop recommendations for the University System of Maryland and Affiliated Institutions (USMAI) Library Consortium in their consideration of several commercial discovery tools to purchase for use in USMAI libraries. Before a product is purchased, it was determined that usability testing should be performed on the four main tools under consideration: EDS, Primo, Summons, and Encore. Dr. Lucy Holman’s IDIA 642 Research Methods class at the University of Baltimore conducted the testing as their final project. To gain insight into each tool’s usability in terms of both negative issues and positive attributes, we conducted usability testing on Summons and Encore using four participants: two graduate students and two faculty members. Participants selected had to be from a university or college within the state university system of Maryland. To conduct the test, we used the Tobii T60 Eyetracker and Software and Morae, a usability testing and market research software, at the University of Baltimore Usability Research Lab. This report concerns the findings of our usability testing. What Are Discovery Tools? Universities within the state university system of Maryland currently use Research Port, developed by the University of Maryland at College Park as a front end for Federated Search. (Federated Search is an information retrieval technology that allows the simultaneous search of multiple searchable resources.) The university system of Maryland is planning to replace Research Port by purchasing a commercial Discovery tool. Currently, when a user logs onto a library website to search for information, he or she faces a daunting task: how to find something doing independent searches on 75 article databases, local digital content on the library server, the library catalog, and electronic collections across libraries.
- 4. Executive Summary (cont.) This is where Discovery tools come in. Discovery tools, which are middleware that is used as a searching front end, make searching faster and easier. By entering a single query, Discovery tools index content simultaneously from a variety of sources such as library catalogs, article databases, and electronic collections. Discovery tools are available as open source, but the most robust ones come from commercial vendors. Summary of Findings In general, our tests showed that Summons was easier to use than Encore. Both the quantitative data and the qualitative data proved that this was true. The quantitative data, which measured such things as the task completion rate, the number of searches needed to complete each task, and the ocular fixations and scan paths, indicate that participants took longer and needed more searches to complete a task in Encore than they did in Summons. The qualitative data, which primarily included an evaluation of participant comments and observations of body language and facial expressions, led us to infer that participants were more confused and frustrated in Encore than they were in Summons. Both tools were lacking in feedback to the user, especially in cases in which a search fails because of a misspelled word. Neither tool offers adequate search hints for frustrated users. However, Encore had many more issues of major or moderate severity, while most of Summons’ issues were minor. Perhaps the most serious error that occurred in Encore had to do with the links to additional content that were located on the right-hand side of the page. Not one user saw these links right away and without clicking on them, search after search failed. The “Results” section explains the findings in more detail.
- 5. Executive Summary (cont.) One of the most surprising findings had to do with one participant’s comments as he was searching. He spoke about how he tries to use the subject matter terms to get to the right Library of Congress catalog, indicating that he applies the traditional thought patterns of a library catalog to his search. Discovery tools are supposed to provide a different search experience for the user than that provided by a traditional library catalog search. This made us question whether the two tools we tested are providing that experience, or whether it is a case of users beginning to change old paradigms as they become more familiar with Discovery tools. Recommendations For most issues, our recommendations include suggestions to reorganize screen elements to make searching less confusing, easier, and faster. Adding more user feedback in the form of search tips and “do you mean” prompts when a word is misspelled would make the user experience less frustrating. In some cases, a simple cosmetic change might make things clearer.
- 6. Participant Overview User 1 User 2 User 3 User 4 Gender Female Female Male Female Age 18-29 18-29 60+ 50-59 UB Affiliation Graduate Student Graduate Student Faculty Faculty Frequency >10 times per 4-10 times per Conducting >5 classes 1-2 classes month month Research Website used first Langsdale library Langsdale library Google Google in research website website Academic Search Academic Search Academic Search Premier; ABI/ Premier; ABI/ Online Databases Premier; Inform; Research Port Inform; Lexis Utilized Psychinfo PyschInfo; Nexis; Research EBSCO; Business Port Source Premier
- 7. Participants (cont.) User 1 User 2 User 3 User 4 -- If none, which -- -- -- -- statement? ** Statement about online #2 #1 #2 #1 databases Online Research Good Good Good Excellent Skills Length of time 1-2 hours 31-60 minutes 0-30 minutes 0-30 minutes researching Frequency 1-3 times per 1-3 times per <1 time per visiting UB <1 time per week week week week library’s website ** Note: The statement numbers, #1 and #2, refer to the appropriate statements on the screeners. Please refer to the screener to read those statements.
- 8. Methodology RECRUITMENT Graduate students and faculty of the University of Baltimore community were contacted about participating in a usability test of two discovery search tools. Team members, as well as Dr. Lucy Holman, used a variety of ways to contact prospective participants, including but not limited to e-mail blasts, individual e-mails, the UB Students Daily Digest, the UB Faculty Daily Digest and word of mouth. A total of four participants who were all affiliated with the University System of Maryland were found to participate in the usability test. Two participants were UB faculty members and two participants were UB graduate students; all were over the age of 18. Participants were tested in sessions that ran approximately 60 minutes each. PREPARATION Prior to the test dates, the team reviewed two discovery tools, Encore and Summons, to determine how each tool worked. A screener, test-script, and consent forms were written and duplicated for each participant. Projects were created in each of the testing software tools, Tobii and Morae. TESTING PROCEDURE Each testing procedure began with introductions and a description of the project. The participant was asked to fill out an informed consent form. A screener was also given to the participant to collect user profile information (as shown in the profile matrix in the previous section). Each of the four participants was asked to perform five tasks in each of the two discovery tools, Encore and Summons. The specific tasks included combinations of searching for articles, books, and audio-visual materials. A Task List is included in the Appendix, pages 48-50.
- 9. Methodology (cont.) Participants were met and welcomed into the reception area of the Usability Lab. The members of the group took turns being the moderator for each participant. Each participant was set up in the testing room and logged into the system. If Tobii was used, it was calibrated to the user’s eye movements. During the test, the moderator followed a written script and emphasized two important points at the beginning and throughout the test: first, that we were testing the discovery tools and not the user or the user’s ability to succeed; and second, that because of the test environment, participants might not be able to fully complete a task. Participants were also encouraged to talk aloud and comment on anything they wanted to during the test. The other two team members sat in the Observation Room behind a one-way mirror. They took notes, observed the users’ actions, and made note of users’ comments. They also recorded any quantitative data. After the test, participants were asked to summarize their reactions to the two Discovery tools and provide us with any additional feedback that could help us in our evaluations.
- 10. Methodology (cont.) CRITERIA OF USABILITY We used the following criteria as a measure of the usability in Encore and Summons: Quantitative Measures Measure of time participants needed to complete each task Task completion rate Number of searches needed to complete each task Average number of searches needed to complete each task Qualitative Measures Conversations with participants after the usability test and answers from the screener questionnaires Comments from participants Facial expressions Body language
- 11. 11 Results Pages 15-24 provide an extensive results overview. Please refer to the Task List, located in the Appendix, pages 48-50, to get acquainted with all of the tasks we asked our users to complete.
- 12. Results Features of Encore and Summons that Support Users Tool Good Usability Practices Current Features of Web Both Tools Allow users to use search criteria in a logical way Left side bar with various search options to further narrow search results produced by a displayed when searching articles in Encore. basic search. This paradigm will fit users’ mental models of the search process.
- 13. Results Features of Encore and Summons that Support Users Tool Good Usability Practices Current Features of Web Summons Allow users to select as much criteria for an Material Type field on the Advanced Search advance search with as little clicks or selections as screen includes an option that combines books possible. This way users will spend less time and journals together. searching and more time looking through meaningful search results. Encore Provide a clean and simple user interface, clear of Encore provides a clean and simple user advertisements, banners, or other added features interface that is fairly intuitive to navigate. that do not provide a purpose in the search process.
- 14. Results Features of Encore and Summons that Support Users Tool Good Usability Practices Current Features of Web Summons Give users plenty of search criteria options in Summons offers a nice Advanced Search form. order to help narrow their search from the beginning.
- 15. Results Error Severity The most critical level - The user was unable to correctly complete Major task. Moderate Significant problems caused for the user. Minor annoyances that slow the user down during the tasks. The user Minor leaves frustrated. Good Indicates that an object is well designed.
- 16. Results Tool Issue Results Level Frequency Encore Users were blind in most cases It took most users anywhere from 5 to 10 Major 4 to finding and/or clicking on minutes to locate this link. One user never “Other Sources” (WorldCat and found it. Until the user located and clicked Link+) “Other Sources,” searches failed. Encore Tool does not clearly show how The user could not understand why certain Moderate 1 user failed it uses search terms results were listed. The results list did not to complete highlight keywords to see why certain data task; was included. Sometimes data was included that did not appear to contain any of the 3 users had search terms. significant problems Encore Irrelevant search results Search results sometimes seemed completely Moderate 2 irrelevant to the search terms entered. The “Did you mean” entries often appeared to be somewhat or very unrelated to what the user was looking for. Encore Tool manipulates keywords It appeared that the tool treated the whole Moderate 4 search string as individual keywords and not specifically as Title, Author, or Subject.
- 17. Results (cont.) Tool Issue Results Level Frequency Both Both tools are unforgiving of When a user misspelled a word, the search Moderate 2 Tools spelling errors. failed. No feedback was provided to bring the misspelling to the user’s attention. Unless the user noticed the misspelling and corrected it, additional searches could also fail. Both Neither tool offers search hints Encore did not provide any assistance or Moderate 4 Tools for frustrated users suggestions to users when a search failed. It did not offer any suggestions as to how the user could get an unmanageable list (too many items) down to a manageable list. Summons offered search tips but they were below the fold of the Advanced Search page and users did not always see them. Both Tool offers multiple ways to Clicking in various search fields produced Moderate 4 tools search for items. search results pages each with a different look/feel and with different filtering options. Navigating through these search options required the user to click multiple times.
- 18. Results (cont.) Tool Issue Results Level Frequency Encore Links for Catalog, Images, and Placement of these links at the very top of Minor 2 Articles are hard to find and the screen made them hard to locate. Their behavior can be confusing behavior is sometimes puzzling. For example, when the user clicks Articles, then clicks Advanced Search, the Articles link disappears and only the Catalog link displays, making the user wonder whether the search is actually being performed on the catalog. Summons Users appeared to hesitate when clicking The tool has two Submit this button as if unsure of which one to use. buttons on the Advanced Minor 3 Some clicked the top button; others clicked Search form. the bottom. Not obvious when the tool Because users did not see the “No entries displayed “No entries found” on found” message, they were unsure whether Summons Minor 2 the top of the Advanced Search the search was still processing or had form. finished. Encore Confusing search results Search displayed “No Catalog Results Minor 1 Found,” at the top of the page; at the bottom of the page, it displayed a whole series of results (in one case over 40,000 articles).
- 19. Results (cont.) Tool Issue Results Level Frequency Summons Not obvious when the tool Because users did not see the “No entries Minor 2 displayed “No entries found” on found” message, they were unsure whether the top of the Advanced Search the search was still processing or had form. finished. Both Screens in both tools were It was not always clear where users should Minor 3 tools cluttered, with options for go when starting a search. searching on the left-hand side, the center, and sometimes on the right-hand side of the page. Both Discovery tools are not Participant tried to use subject matter terms Minor 1 tools providing a different search to get to the right Library of Congress experience catalog, applying the traditional thought patterns of a library catalog to his search. Discovery tools are supposed to provide a different search experience for the user than that provided by a traditional library catalog search.
- 20. Results Number of searches for each task Encore Summons Encore Summons User 1 User 2 T1 6 1 T1 6 2 T2 7 1 T2 6 2 T3 4 2 T3 4 2 T4 4 1 T4 5 1 Encore Summons Encore Summons User 3 User 4 T1 10 1 T1 7 2 T2 10 2 T2 5 1 T3 9 6 T3 6 4 T4 3 2 T4 10 1 Overall, users had the most problem with successfully completing task 1 in Encore and task 3 in Summons
- 21. Results How successful were users in completing tasks? Participant completed all of the components of a certain task successfully. Complete Example: Participant found many articles on Artificial Intelligence that were no more than five years old and are peer-reviewed. Participant only completed some of the components of a certain task. Incomplete Example: Participant successfully found an article but did not save an article for retrieval. Participant was not able to complete any portion of the task. Failed Example: Participant did not find a known item: Beowulf
- 22. Results How successful were users in completing tasks? Failed Tasks 15% Failed Tasks 35% Completed Tasks 45% Incomplete Tasks Completed Tasks 25% 60% Incomplete Tasks 20% Encore Summons
- 23. Results Average Time on Task in Encore and Summons Based on our findings, Task 2 in Encore gave users the most problem. It took an average of 7:50 minutes for users to complete the task or end the task before asking to move to the next task. Similarly, it took users an average of 5:15 minutes to complete or end Task 3 in Summons.
- 24. Results Time on Task in Encore and Summons User 1 User 2 User 3 User 4 Encore Task 1 4:31 2:44 5:40 2:51 Task 2 7:39 5:29 14:25 3:49 Task 3 1:30 3:49 9:50 4:45 Task 4 4:43 4:39 4:31 3:27 Task 5 0:14 Summons Task 1 0:42 0:48 1:44 2:44 Task 2 0:40 0:42 4:03 0:50 Task 3 2:41 2:40 7:18 8:21 Task 4 1:29 2:14 6:16 2:22 Task 5 In general, it took users much more time to complete or end tasks in Encore than in Summons. Interestingly, Task 2 in Encore gave users the most problem, as their completion times are higher. In Summons, users were much more successful in completing tasks. Interestingly, Task 3 gave users the most trouble with time rates varying between 2:40 - 8:21 minutes, however these times do not compare to the problems Task 2 gave users in Encore. Comparing Task 2 in Encore and Summons, users cut their time rate by more than half.
- 25. Interesting Observations The observations described below include those things that we felt were important or interesting enough to make a note of while we were conducting our testing. • Only one participant found the + button in the Format field of the Advanced Search screen in Encore. • Only one participant typed in “peer reviewed” as a search term in the advanced search criteria. The other participants were unsuccessful in filtering articles by this criteria because there was no specific option and it didn’t occur to them to include it as a search term. • One participant had no successful searches in Encore because this participant never located the “Get More Results in: Link+ and WorldCat” on the right-hand side of the screen. It appeared from our observations that the right-hand side of the screen was totally ignored during the usability testing for this tool with this participant. • Only one participant was successful in saving the article for retrieval in Task 5. The other participants found the article but did not save. • Participants who used a basic search and then narrowed their search results using additional search criteria seemed to have better results with both tools (provided that they had already clicked Link+ or WorldCat in Encore) than those who started with Advanced Search. One of the most surprising findings we came across in our usability testing had to do with one participant’s comments as he was searching. He spoke about how he tries to use the subject matter terms to get to the right Library of Congress catalog, indicating that he applies the traditional thought patterns of a library catalog to his search. Discovery tools are supposed to provide a different search experience for the user than that provided by a traditional library catalog search. This made us question whether the two tools we tested are providing that experience.
- 26. Recommendations Encore Recommendations Highlight words in results page to show user how terms are being used. All results should appear below the primary search field. Use better labeling such as “Get More Results in: Link+ and WorldCat.” If no search results are found, this should be located where the results would be. If results are found, move this to the top left sidebar where there is a higher probability a user would look for more information on the page. Summons Recommendations Have only one Submit button on Advance Search Form page. Make the words “No Entries Found” in a larger font size and in the color red, located below the search fields where results would normally appear, so that user has a clear understanding that their search returned zero results and that the search has completed. Common Recommendations Have a Modify Search Link below the search field to edit search or make changes to the search directly from the Search Results page. Have a list of the top search term suggestions. Spelling suggestions are needed so that a search does not always result in “No Entries Found”. Clear call-out button for help or instructions on how to use the discovery tool. Perhaps a video tutorial or PDF guide on how to use the system. Help link/button should also be clearly labeled on all internal web pages.
- 27. Overall Recommendation Should a purchasing decision need to be made between Encore and Summons, our group feels that Summons would be the better discovery search tool for the graduate students and faculty members of University of Baltimore, and as a whole for the University System of Maryland and Affiliated Institutions (USMAI). While some changes should be made to Summons before it is integrated into USMAI, we feel that these changes are minor in comparison to the issues Encore would need to address. The results show that many users felt Summons was a lot easier to use compared to Encore. Users successfully completed 60% of the five tasks in Summons versus users only completing 45% of the tasks in Encore. The overall failure rate of successfully completing tasks shows an even greater disparity; users testing Summons only had a 15% failure rate compared to Encore’s 45% failure rate. We feel that if our recommendations were taken into consideration, the failure rate in Summons would drop significantly.
- 28. 28 Appendices
- 29. 29 Heat Map Heat Maps show how long each part of the screen has been looked at, as well as the main areas of the website users looked at and focused on.
- 30. Heat-Maps These heat maps show that users have a right-side “blindness” in both Encore and Summons. In other words, most users did not notice that there was a portion on the right side bar in Encore that said “Other Sources”. We assume this “blindness” occurs because advertisements and other banners normally appear on the right hand side on webpages. On average, it took participants approximately 1:34 minutes to find “Other Sources” in Encore. User 4 in particular never even found the “Other Sources” side bar. A similar pattern of “blindness” was also found in Summons. For example, none of our participants looked to the right sidebar in Summons to see what features or links were available. User 2 even commented that her field of vision was in the center of the screen and to the left.
- 31. 31 Wireframes
- 32. Wireframes The following wireframes were created in response to our recommendations, based on both of the discovery search tools we tested. We envisioned a discovery search tool that incorporates all of our recommendations. Our wireframes also include the positive features we found in both systems.
- 34. Wireframes
- 35. Wireframes
- 36. Wireframes
- 37. 37 Consent Form
- 39. 39 Faculty Screener
- 40. Faculty Screener 1.! What!is!your!age? !!18!–!29!!!!! !!!30!–!39!!!!! !!40!–!49!!!!! !!50!–!59!!!!! !!60+ ! 2.! How!often!do!you!conduct!research,!for!classes!or!personal!interest?! !!Never! !!Rarely! !!Two!or!three!times!per!month! ! !!Four!to!ten!times!per!month! !!More!than!ten!times!per!month ! 3.! When!first!conducting!research,!which!of!the!following!do!you!typically!go!to!first?! !!School!library!website!for!resources !!Online!Search!Engine!(please!specify)!!_______________ !!Other!!___________________________ ! 4.! Which!online!databases!do!you!use?!(Check!all!that!apply)! !!Academic!Search!Premier! !!ABI/Inform!!!!! !!MasterFile!Premier! ! !!PsychInfo! !!Ebsco! !!Other:!!_______________________ _____!!None/I!do!not!use!online!databases!to!search!for!information! ! 5.! If!you!selected!“none”!!from!question!4!above,!what!is!your!reasoning?! !!I!never!find!what!I’m!looking!for.!!!! ! !!It!is!too!difficult!to!use.! ! !!Other!(Please!Specify):!!___________________ ! 6.! If!you!use!the!library's!search!tools,!which!statement!best!reflects!how!you!feel!about!your!use!of! online!database!tools:! !!I!can!locate!all!of!the!appropriate!resources.! ! !!I!am!missing!some!of!the!resources!available!to!me.! ! !!I!am!not!able!to!use!the!search!tools!to!find!what!I!need. !!I!was!not!taught!how!to!use!these!tools. ! 7.! How!would!you!rate!your!searching!skills?! !!Poor! !!Fair! !!Good! !!Excellent ! 8.! On!average,!for!how!long!do!you!do!research!in!one!sitting?! !0!to!30!minutes!!!!! !!31!to!60!minutes!!!!! !1!to!2!hours!!!!! !More!than!2!hours ! 9.! How!often!do!you!visit!your!library’s!website!to!locate!resources?! !!2+/day!!!!! !!1/day!!!!! !!4+/wk!!!!! !!1"3/wk!!!!! !!less!than!1/week!!!!! !!Never
- 41. 41 Graduate Student Screener
- 42. Graduate Student Screener 1.! What!is!your!age? !!18!–!29!!!!! !!30!–!39!!!!! !!40!–!49!!!!! !!50!–!59!!!!! !!60+ 2.! What!is!your!major!and!concentration?! Major:!__________________________________! Concentration:!________________________________! 3.! In!your!academic!career,!how!many!classes!have!required!you!to!conduct!research?! !!0!!classes!!!!! !!1"2!!classes!!!!! !!3"4!classes!!!!! !!More!than!5!classes 4a.! When!first!conducting!research,!which!of!the!following!do!you!typically!go!to!first?! !!School!library!website!for!resources!!!!! !!Online!Search!Engine 4b.! If!you!selected!Online!Search!Engine!in!Question!4a.!what!search!engines!do!you!normally!use! for!conducting!research?!(Check!all!that!apply)! !!Google! !!Yahoo!! !!Bing/MSN! !!AskJeeves! !!AltaVista !!Other!(Please!Specify):!!________________ 5a.! Which!online!databases!do!you!use?!(Check!all!that!apply)! !!Academic!Search!Premier!!!!! !!ABI/Inform!!!!! !!MasterFile!Premier!!!!! ! !!PsychInfo! !!Ebsco! !!Other:!!___________________ !!None/I!not!use!online!databases!to!search!for!information 5b.! If!you!selected!none!from!question!5a!above,!what!is!your!reasoning?! !!I!never!find!what!I’m!looking!for.!!!! ! !!It!is!too!difficult!to!use.! ! !!Other!(Please!Specify):!!___________________ 6.!!!!If!you!use!the!library's!search!tools,!which!statement!best!reflects!how!you!feel!about!your!use!of! online!database!tools:! !!I!can!locate!all!of!the!appropriate!resources.! ! !!I!am!missing!some!of!the!resources!available!to!me.! ! !!I!am!not!able!to!use!the!search!tools!to!find!what!I!need. !!I!was!not!taught!how!to!use!these!tools. 7.! How!would!you!rate!your!searching!skills?! !!Poor! !!Fair!!!!!!!! !!Good!! !!Excellent 8.! On!average,!for!how!long!do!you!do!research!in!one!sitting?! !!0!to!30!minutes!!!!! !!31!to!60!minutes!!!!! !!1!to!2!hours!!!!! !!More!than!2!hours!! 9.! How!often!do!you!visit!your!library’s!website!to!locate!resources?! !!2+/day!!!!! !!1/day!!!!! !!4+/wk!!!!! !!1"3/wk!!!!! !!less!than!1/week!!!!! !!Never
- 43. 43 Script
- 44. Script Introduction M: Good <time of day>, thank you for making time in your schedule to participate today. My name is <name>, and I am a part of a group working on a research study to learn about discovery (search) tools for the University System of Maryland and Affiliated Universities (USMAI). We have asked you here <time of day> to help us evaluate two discovery tools through your interaction with them. At a basic level, a discovery tool is library search engine that allows you to search multiple databases, along with the library's catalog in one search interface. During today’s session, you will be using <name discovery tool> and <name discovery tool> to complete some basic tasks that a <student/faculty member> might carry out during their search process. Please realize that we are not testing your ability. Instead, we’re testing the effectiveness of the tool. Do you feel comfortable in proceeding? P: Participant acknowledges M: Very good, if you will, please sign our release form here. It says: “I, the undersigned, agree to be part of a usability study conducted at the University of Baltimore. As a participant. I agree to be videotaped and to have my activities on the computer recorded. I allow my comments and observations about my experiences to be become part of the findings of the usability study.” P: Participant signs release form. M: Thank you. Before we begin with the usability test, we could like you to fill out this basic questionnaire in order to get an idea of your background in searching online databases and online search engines for resources such as books, articles, or audiovisual items. (Hands the participant the screener). P: Participant fills out screener and hands it back to the moderator. M: Thank you.
- 45. Script (cont.) System Introduction and Calibration (Tobii) M: In order to start the usability test we need to set up the eye-tracking software. This computer system is equipped with eye- tracking software, which is a device for measuring eye positions and eye movement which is helpful in analyzing how you use the discovery tools. Before we begin, we will need to set up a session in the software. Would it be OK if I used your first name to identify this session? P: Participant responds. M: O.K. <name of user>. The next step is to calibrate the eye tracker to your eyes. During this calibration and throughout this session, please relax and continue to look at the screen. While we will be conversing throughout the session, please try to keep your focus on the screen rather than looking at me. Do you have any questions? P: Participant responds. M: OK, here we go… (start and run through calibration) FAILURE M: OK, the system had a little trouble picking up all the information it needed for calibration. I would like to recalibrate the areas of the screen on which the system had some trouble. Are you ready? P: Participant responds M: OK – Here we go SUCCESS M: Excellent! I will now open the first discovery tool.
- 46. Script (cont.) Alternate paragraph if using Morae: (Will the Morae session be set up prior to beginning the test, or will we have to set it up before the user starts working with the tool?) M: The computer system you will be working on today is equipped with Morae software, which is used for usability testing and user experience research. It records your interactions with the computer so they can be analyzed after your session. I’m now going to set up a session in the software. Would it be OK if I used your first name to identify this session? P: Participant Agrees Familiarization M: Here is the first discovery tool you will be working with. As you can see, <name of tool> includes options that allow you to search library databases for books, articles, and other types of materials. Overview of Tasks M: To gauge the usability of the tool, we will be asking you to work your way through some tasks that we would expect a <student or faculty member> to do. These will be tasks that you may or may not have done before. Because of the test environment, you may not be able to FULLY complete a task and may be stopped at the point of a login or click a link that is unavailable. If you can’t find something or if it doesn’t make sense, just tell us, and we can inform librarians who are making decisions about the tools. Remember: We are testing the discovery tools, not you. There is no right or wrong answers, no right or wrong way to do things. Every action you take, no matter how you may feel about how it turns out, helps us to evaluate the product. Do you have any questions before we begin? P: Participant responds
- 47. Script (cont.) Tasks – Discovery Tool 1 Insert Discovery Tool Tasks and Scenarios. Tasks- Discovery Tool 2 M: Here is the second discovery tool that you will be working with. As you can see, <name of tool> includes options that allow you to search library databases for books, articles, and other types of materials. To gauge the usability of this tool, we will be asking you to work your way through the same tasks that you did using the previous tool. Again, because of the test environment, you may not be able to FULLY complete a task and may be stopped at the point of a login or click a link that is unavailable. If you can’t find something or if it doesn’t make sense, just let us know. Do you have any questions before we begin? P: Participant responds Insert Discovery Tool Tasks and Scenarios. Conclusion M: Those are all of the tasks we have today. Thank you for your assistance in evaluating the discovery tools; we appreciate your time completing them for us. Your participation today will help us make recommendations that can make a difference to the UB community. Do you have any further questions about the study before you leave today? P: Participant responds. M: (Answers questions if any are asked). Once again, thank you for spending your (morning/afternoon) with us today. As a thank you, please accept this gift of appreciation. (Hand over $10 Starbucks gift card)
- 48. 48 Task List
- 49. Task List Task Script 1. Find Your professor has asked you to read the Seamus Heaney translation of Beowulf. Find out if a copy is known item available in the library and, if so, where it is located. If a copy is NOT available, to check out, are (book) there any other options for obtaining or accessing the book? Alternate wording for faculty: You are interested in reading Seamus Heaney’s translation of Beowulf. Find out if a copy is available in the library and, if so, where it is located. If a copy is NOT available, to check out, are there any other options for obtaining or accessing the book? 2. Find Your professor suggested that you consult an article entitled “Modernism and the Harlem Renaissance” that known item was published in American Quarterly for a paper you’re doing. You want to see if it’s available full-text from the (article) & save library’s databases. Once you have found the article, can you save that information for later use? article citation Alternate wording for faculty: There is an article you recall seeing in American Quarterly on “Modernism and the Harlem Renaissance” that you think you might assign as a class reading. You want to see if it’s available full-text from the library’s databases. Once you have found the article, can you save the information to pass on to your students? (Full citation, do not give to subject: Baker, Houston A. (1987). Modernism and the Harlem Renaissance. American Quarterly, 39 (1), 84-97.) 3. Conduct You are interested in learning basic Portuguese for an upcoming trip to Brazil. You want to see if the library has topic search for any elementary Portuguese language textbooks or audiovisual materials print or AV materials
- 50. Task List Task Script 4. Conduct a You have to research Artificial Intelligence for a project, and your professor has asked you to use a variety of topic search for resources including books and journal articles. The articles should be no more than five years old and be peer- all types of reviewed. As a preliminary assignment she wants you to report on how many relevant books and articles you materials found. Alternate wording for faculty: You are researching recent developments in Artificial Intelligence. As a preliminary step, you want to compile a list of relevant books and peer-reviewed articles from the last five years that are available through your library. 5. Retrieve article You’re ready to use the Modernism article you found earlier, and you want to retrieve it.