VuFind @ Illinois #4 Usability Testing and Results
- 1. VuFind@Illinois: Usability Testing & Results
- 2. Usability & Feedback: An Overview Asked for Feedback Survey Formal Usability testing
- 3. Feedback
- 4. Feedback: Comments Love it love it love it! MUCH more user friendly. I don’t understand what “barcode” is for requesting items. How do you set up an account? Can it be used to search I-Share? Where is call number searching? I like the facets, but want more facets Is reserves information in VuFind?
- 5. Feedback Needed more data Began to think about formal usability testing Then, Yale did a survey…
- 6. Survey: Results Types of Search Known Item 31 Keyword/Subject Search 48 Books 29 Journals 5 Multimedia 8 Articles 5 Author 12 Course Reserves 2 Call Number 1
- 7. Survey: Results 75% Liked VuFind 86% Found what they were looking for 73% Thought the search results were what they expected 54% Used the facets 63% Thought facets were helpful
- 8. Survey: Facets “ I didn’t even really pay attention to them. They were not generally helpful in the kind of search I was doing.” “ I expected to find articles when clicking journals/magazines, like WorldCat.” “… because VuFind has the look and feel of Amazon, I only focused on the book listings, choosing instead to ignore the screen areas usually laden with advertising.”
- 9. Survey: Facets “ They really helped pare down the tidal wave of results I initially got.” “… they were very helpful in making my search more specific since there aren’t a lot of default search options.” “ I think it helped by narrowing down the results, sort of like Google does these days.”
- 10. Survey: Facets “ The results were not what I expected, but it was a pleasant surprise. I found things in my research that I didn’t know we had.” “… the most useful part of the groupings for me was helping me sort of mentally focus what I was looking for.”
- 11. Survey: Results In the future, if you need to find books, the first place you’d start is: WebVoyage 30% WorldCat 8% Amazon 4% Google 12% VuFind 41% Other 7%
- 12. Survey: Results I think that I would like to use VuFind frequently: Strongly Disagree 3% Disagree 12% Neutral 19% Agree 52% Strongly Agree 14%
- 13. Survey: Results I found VuFind unnecessarily complex: Strongly Disagree 28% Disagree 50% Neutral 13% Agree 8% Strongly Agree 1%
- 14. Survey: Results I thought the system was easy to use: Strongly Disagree 0% Disagree 6% Neutral 11% Agree 56% Strongly Agree 27%
- 15. Survey: Results I felt very confident using VuFind: Strongly Disagree 3% Disagree 10% Neutral 22% Agree 43% Strongly Agree 23%
- 16. Survey: Comments - Pluses Not very overwhelming & user friendly Easy to navigate and narrow results Liked cover images Interface more user friendly than current catalog Similar interface to area public libraries Not as intimidating as WebVoyage
- 17. Survey: Comments - Improvements Did think results were well sorted and not clear how it does sort the results Facet and “facets now applied” are confusing terms Text formatting and displayed information is sometimes confusing Could use some more polish Too much scrolling – very vertical
- 18. Survey: Comments - Improvements Users want to use it to search for articles Want even more information about an item Not sure of relevancy of user added content Some facets are confusing No ability to sort by a particular campus library
- 19. Survey: Comments – Feature Requests Direct export to RefWorks Keyword search option Call number searches Social networking integration Have a “Repeat Search in I-Share” link in record Nice to customize display to show what you wanted on results page Ability to collapse facets
- 20. Survey: Comments – I-Share Perception interface with I-Share needs to be clarified and streamlined Not clear how to conduct search in I-Share Liked not inputting library card number multiple times requesting items Confusion over why you need to create an account
- 21. Survey: Comments “ Please replace the current system with VuFind right now, I really don’t see a reason not to. It is a huge improvement” “ VuFind is going to make the old online catalog as awkward as using the card catalog is now” “ VuFind strikes me as a dumbed down version of the Google search page designed to appeal to freshmen who are used to doing Google searches rather than a serious catalog geared towards scholars conducting serious research.”
- 22. Usability Testing - Setup In April & May, 2009, conducted 15 hands on usability tests with Morae 5 faculty, 5 graduate students, 5 undergrads Very diverse group Used task based test adapted from statewide WorldCatLocal usability testing
- 23. Usability Testing: Instrument Asked general interface/layout questions and if search and features are clear Searches Broad subject search for “climate change” Search for James Joyce Prince audio recording “ Gone with the Wind” DVD “ Jack London” and “White Fang” Find a public speaking DVD
- 24. Usability Testing: Results Lack of keyword searching
- 25. Usability Testing: Results Language Selection
- 26. Usability Testing: Results Red Light/Green Light for holdings—need yellow light for reference or reserve items
- 27. Usability Testing: Results Direct Export to RefWorks:
- 28. Usability Testing: Results Want URLs to stand out more:
- 29. Usability Testing: Results Ability to link out to I-Share VuFind Example:
- 30. Usability Testing: Results Favorites feature:
- 31. Usability Testing: Results Facets:
- 33. Usability Testing: Results Format
- 34. Usability Testing: Results Topic/Subject Area
- 38. Usability Testing: Results Staff view
- 39. Usability Testing: Results Tagging
- 40. Usability Testing: Results Interface improvements: Could use more polish Too vertical Ability to customize number of results on a page Change order of facets in sidebar Make electronic resources/ebooks a clearer limit in the advanced search and facets Searching for multimedia was cumbersome
- 41. Usability Testing: Results Really liked how visual the interface was—less text, more pictures Users care less about “librarian features” such as MARC and call number searching Users aren’t familiar with next generation catalogs or other library tools, but use online bookstores
- 42. Usability Testing: Results All but one subject thought VuFind was best for most of their catalog use Only 1 user preferred layout of Amazon more, citing the star ratings All will use VuFind again Everyone thought it was a dramatic improvement to WebVoyage
- 43. Summary VuFind isn’t perfect, but is a huge improvement over WebVoyage.
Editor's Notes
- #3: Did both quantitative and qualitative evaluation.
- #4: Got some great early feedback; not a lot, but it did come Feedback email went to all staff involved with next generation catalogs Feedback became a part of our “summary of issues” we communicated periodically with Carli Comments were mixed: a few told us they loved VuFind, some hated, most were recommendations for improvements and features
- #5: These are so general; wanted more specific feedback
- #6: We asked nicely and Yale gave us their survey, so we decided to repeat it. We wanted our own data with little effort and we thought it would be nice to compare with Yale Yale’s survey asked users if they had completed a search in VuFind and what they thought of both VuFind and the search results
- #7: 142 Participants
- #9: Survey results on facets were more mixed than we expected. But comments told us users liked facets, but often found them frustrating.
- #12: Comment: 9 Others: At least 2 of “other” stated easy search; two said they’d use both Voyager & VuFind; 1 iShare; 1 used Chicago Public Library (??) Thought VuFind easier to navigate, but Voyager better for requesting from other libraries 1 Commenter said “You should look at Aquabrowser”
- #19: They want that ubiquitous single search box Not sure what they wanted when they wanted “more information” Relevancy of user added content includes tags, reviews
- #25: Confusion of what an all fields search was – users understood term keyword more often
- #26: Why not pull out all languages in catalog on advanced screen? But you can always facet later
- #27: Really liked this feature being on results page, but needed an intermediary for reserve or reference items
- #28: No direct export!
- #29: Title is not a URL, author is—uses same color
- #31: Loves feature, but…. Users confused about close and save in adding to favorites. Also requested favorites list to be more robust and customizable.
- #32: Everyone liked facets because they helped you to discover items, but there were issues with them. Idea of facets (and users realized this) is to refine after you search, not having to refine before you conduct your search; they liked this
- #33: Many complaints about looking up items in I-Share libraries, but there were no good solutions to this problem
- #34: A lot of confusion over formats. Wanted to further limit multimedia to specific format (DVD, CD, etc) instead of film/video and music recording This limitation caused non-print items to be nearly impossible to find
- #35: Confusion over what exactly these are; They come from the bib record Topic is the subject heading Subject area is from the classification code Genre coming from 655 – Index term/form
- #36: Era was really confusing because there was no standard cataloging Also, user expected this to be the date of publication or copyright, not the era the item is about—really hard for multimedia
- #37: Why is author so futzy? No matter how you input James Joyce, this is what you get. Cataloging issues, Also pulls out items in which name is “other author”
- #38: Comments/reviews Users thought these would be helpful, but wouldn’t add them themselves. Several users questioned authority of these and thought they could be biased or even incorrect. Wanted reviews only from authoritative sources (not Amazon, but maybe Choice or LJ) Several users didn’t like them at all, saying if they wanted reviews, they’d go to Amazon.
- #39: All users hated staff view and wanted it to be less prominant
- #40: About half of users in all areas knew about tagging, and generally liked the concept. But only one said they’d actually add tags. Useful applications for tags that were mentioned include specific media format (DVD) and foreign language title translation
- #42: Library tools included iRead and LibraryThing
- #43: 1 subject was a GSLIS prof—missed call number search
- #44: Everyone wanted to know if it will stay