Providing Access to Digitized Content Via the Finding Aid: A Usability Study

Editor's Notes

• #2: Hi, I’m Jody DeRidder from the University of Alabama Libraries. I’m here to talk about a usability study on access to digitized content via the finding aid, without hand-created item-level metadata.
• #3: In this time of reduced funding and increased demand for access to primary source materials online, a number of institutions have turned to online access to digitized items via the finding aid. The intent is to leverage the EAD descriptions for search and retrieval, while simulating the user experience in the reading room. Content is presented in the order encountered in the boxes and folders. This approach enables low-cost digitization of even large manuscript collections, providing online access to material that otherwise may never be digitized.  In a grant project partially funded by the NHPRC, the University of Alabama Libraries developed a low-cost model and supporting open-source software for implementation. The grant project included a usability test which compared the resulting interface to a similar collection delivered with item-level descriptions accessed outside the finding aid.  At an estimated cost of 79.5 cents per page, our mass-digitization method costs less than a third of our usual item-level description access. However, we needed to know how useful this interface is for our patrons.
• #4: We sought to measure efficiency, effectiveness, satisfaction and the critical element of learnability.Efficiency was defined by the two measures of time on task and total number of steps (clicks) required for a participant to successfully complete a task. We counted double clicks as single clicks, and a string of text entered as a single interaction. Effectiveness was measured by whether or not a task was completed successfully. We measured satisfaction by both the users’ overall perceived difficulty of each interface (on a 1-5 scale), as well as the total number of positive versus negative comments about an interface recorded during testing. Negative comments were assigned a value of -1 and positive comments assigned a value of +1; duplicate comments by the same user were disregarded.For learnability, we examined improvements from task 1 to 4 for both interfaces, in the following variables: time to first click (which may indicate indecision), total time to locate content, number of steps (as defined above), and success in completing a task.
• #5: Experienced researchers have already been shown to prefer the finding aid as interface. Both Scheir and Chapman found that novice users experienced a learning curve during exposure to finding aids, gaining confidence and ease with time. For our study, participants were primarily novice users.This study included twenty participants: 8 undergraduate students, 10 graduate students, one post-graduate volunteer and one college-educated staff member. Subsets of these participants included those with and without digital library experience, special collections experience, background in history, and English as a second language.
• #6: The test consisted of 4 known-item searches, repeated for each of two similar collections accessible in different ways. The item-level described digitized items in the Jemison collection could be searched using a search box, while the Cabaniss items were accessible via links embedded in the EAD finding aid. The order in which we presented collections to participants was alternated so that one collection would not always benefit from the participant’s experience with the previous collection.Prior to performing the tasks in each collection, participants were given short introductions to both the collection and the interface. The Jemison collection was presented as a result list of items with the search options set to isolate queries to this collection.To locate items within the Cabaniss collection, participants had to navigate the EAD finding aid, as no “search within page” option was available. Participants with no previous experience with special collections were told that a finding aid is a guide to the collection created by the archivists, and, in this instance at least, is similar to a table of contents for the collection. Morae software was used to record the sessions. One researcher sat with the participant and provided verbal directions while another observed and captured data. After completing the tasks, participants were asked to rank the interfaces and to comment on their experiences and preferences. After completing the survey, participants were given a 1 GB flash drive in exchange for their assistance.
• #7:  Overall, participants required an average of 35% less time and 48% fewer interactions with the item-level described collection than with the finding aid as web interface. This indication of reduced efficiency is to be expected without a search option in the finding aid page, as results may only be obtained via browsing. Success rates via the item-level search interface were 7.5% higher. Participants as a whole clearly prefer the item-level interface, by a factor of 3:1, though 40% of those with a background in history, and a third of those with special collections experience or without digital library experience preferred the finding aid interface.
• #8: Marked differences in efficiency were evident in both interfaces for participants for whom English is a second language, but the difficulty was more pronounced in the finding aid interface (51% more time and 10% less success in the item-level interface; 41% more time and 13% less success in the finding aid interface). 80% of the participants for whom English is a second language preferred the item-level interface. Interestingly, those without previous digital collection experience found the finding aid interface significantly easier than those who claimed familiarity with the more traditional digital library interface. The EAD interface took them 42% less time, 27% fewer clicks, and provided 12% more success.This bodes well for future acceptance of this method of web delivery, and corroborates Chapman’s findings that “groups that showed the most significant improvement over time were novice participants and Internet users with a beginning proficiency level.”
• #9: For the learnability interface comparisons, we reduced the sample to 14 by removing six users from the Jemison-First group. This was necessary to control for an experience effect in which some aspect of the first interface encountered could impact performance in the second. Unfortunately, the resulting sample contained three less non-native speakers. If indeed an interface is more learnable, we would expect to see statistically significant improvements in effectiveness and efficiency from task 1 to 4 for all users in each interface separately. Although improvement from task 1 to 4 did occur 62.5% of the time in favor of Jemison, these differences were not statistically significant .
• #10: We need more tests on the finding aid interface to determine what actually helps users. Suggestions from the research include: • replacing archival terminology• Providing search in page feature• Providing navigation links for sections of the finding aid on the leftTHEN: we need learnability tests for novice users that span multiple sessions. Tullis and Albert make an excellent argument for comparing multiple sessions with the same participants, capturing the same metrics, as this would more closely simulate real-life experience.
• #11: Finding aids present digital materials in the context of the collection, and hence provide far more information to be sifted than content described solely on the item level. Efficiency and effectiveness measures should not be applied in comparing the EAD interface with item-described content. The result is a comparison of apples and oranges.What is truly at issue here is learnability, particularly for novice users and those for whom English is a second language. Modifications to the display and terminology should be tested to verify that these changes increase access and learnability.By increasing the ease of use and verifying the learnability of the finding aid interface, we will be better positioned to leverage this low-cost digitization method to provide online access to large manuscript collections.
• #12: I’ve included a bibliography for the research to which I’ve referred, as well as links to our wiki, project site and display. An article about this project has been submitted to American Archivist and hopefully will soon be available there.