“From Discovery to Fulfillment: Improving the User Experience at Every Stage.”

Editor's Notes

• #3: Image https://www.flickr.com/photos/lucasartoni/14941319645 by Luca Sartoni CC BY-SA 2.0
• #5: Jay Where is the content that we could throw these documents into? Caro? Took all user journeys and mapped them out onto this board to see what conceptual model could fit the workflows. This is what we’ve come up with. Informed by Clutch research. See https://insideoclc.sharepoint.com/:b:/s/products/GPM/Operations/EZhEKZvwlpxNuMcMw0gSk8UBR3ANgUUw4jCeOUhj9VALqg?e=yasZQy See also core discovery user journeys: https://insideoclc.sharepoint.com/:w:/s/products/GPM/Operations/Ech0Yj1d4VlNi7y5Hxz6iwEBMgDGZije13zB3nBi3mFBhw?e=PNA8wI
• #7: Image: https://www.flickr.com/photos/75487768@N04/6786332997 by barnyz CC BY-NC-ND 2.0 Lynn Add heading – show that this is the scope of this study more succinctly. OCLC embarked on a research project to identify how academic library users navigate the path from discovery to access. The project analyzed all WorldCat Discovery search logs from April 2018 in aggregate and analyzed search logs from specific users combined with interviews with those users. Research Questions Last Updated February 1, 2018 1. How do users navigate the path from discovery to access?      a. Where do people leave the system/discovery?      b. How long are user sessions?      c. With what features do the different types of academic library users interact? 2. What do academic users do when searches don’t result in fulfillment?      a. Go elsewhere for same thing?           i. Where?          ii. Why?      b. Get something different here instead? 3. What differentiates searches that lead to access from searches that don’t lead to access?      a. How many and what percentages of searches lead to access (are delivered/accessed)?      b. How many search terms are used for those searches that lead to access?      c. Do people refine their searches, if they do not find something of interest to them? If so, how do they refine their searches?      d. Does search refinement affect access rates? If so, what is the access rate after search refinement and does that differ between the access rate for searches not refined? 4. What demographic characteristics (personal and institutional) influence the access of users?      a. How are small academic users versus large ARL users using the system the same and differently? 5. To what extent does access to sources correlate/define a successful or unsuccessful search?
• #8: Image https://www.flickr.com/photos/75487768@N04/6786397239 by barnyz CC BY-NC-ND 2.0 Lynn Jansen, Bernard J. 2006. “Search Log Analysis: What It Is, What’s Been Done, How to Do It.” Library and Information Science Research 28, no. 3: 407–432.      Connaway, Lynn Silipigni, and Marie L. Radford. 2017. Research Methods in Library and Information Science, 6th ed. Santa Barbara, CA: Libraries Unlimited.    Hunter, Rhonda N. 1991. “Successes and Failures of Patrons Searching the Online Catalog at a Large Academic Library: A Transaction Log Analysis.” RQ 30, no. 3: 395–402.  https://www.jstor.org/stable/25828813.     Jamali, Hamid R., David Nicholas, and Paul Huntington. 2005. “The Use and Users of Scholarly E-Journals: A Review of Log Analysis Studies.” Aslib Proceedings 57, no. 5: 554–571. https://doi.org/10.1108/00012530510634271.     Lamkhede, Sudarshan, and Sudeep Das. 2019. “Challenges in Search on Streaming Services: Netflix Case Study.” In the Proceedings of SIGIR ’19, July 21–25, 2019, Paris, France. https://arxiv.org/pdf/1903.04638.pdf.     Nouvellet, Adrien, Florence D’Alché-Buc, Valérie Baudouin, Christophe Prieur, and François Roueff. 2019. “A Quantitative Analysis of Digital Library User Behaviour Based on Access Logs.” Qualitative and Quantitative Methods in Libraries 7, no.  1: 1–13.  
• #9: Image https://www.flickr.com/photos/75487768@N04/7598096750 by barnyz Lynn Connaway, Lynn Silipigni, John M. Budd, and Thomas R. Kochtanek. 1995. “An Investigation of the Use of an Online Catalog: User Characteristics and Transaction Log Analysis.” Library Resources and Technical Services 39, no. 2: 142–152.  
• #10: Brittany We started by analyzing the data that was already available to us—search logs from WorldCat Discovery. Benefit: “One gets real behaviors from real users using real systems interacting with real information. Log analysis is everything that a lab study is not.” (Jansen 2017, 349) Caveat: Log data is trace data, meaning it is a trace left behind of what a user did, and it must be interpreted. Jansen, Bernard J. 2017. “Log Analysis.” In Research Methods for Library and Information Science, 6th ed., edited by Lynn Silipigni Connaway and Marie L. Radford, 348-349. Santa Barbara, CA: Libraries Unlimited. 
• #11: Brittany One of the first things that we had to establish was what the search logs could tell us about user behavior. So we looked manually at some logs to determine what type of information we could glean. The logs could tell us: Type of search (index) Search terms/string Elements of the system interacted with, in this case the autosuggest feature Evolution of searches in a search session Part of this process was working with libraries to recruit participants so we could connect logs with the users who created them. Logs in the system are anonymous—they can’t be tied to the user who generated them. The participants we recruited gave us the ID from their search, which we then used to pull all of the log events associated with that search. On the left-hand side of this slide you can (maybe) see the search log for this participant, CBU09. What we found in interpreting the logs was that this participant started with a mistyped keyword search that returned 0 results. They corrected their spelling and returned over 700,000 results. They then began typing an additional word into their keyword search and selected one of the autosuggested keyword phrases, returning just under 2,000 results. Once we had interpreted the participant logs, we used that information to build an interview protocol. Peggy will talk about the interview process a little later. CBU09 - Female, 19-25, undergraduate, humanities, Christian Brothers University
• #12: Peggy As mentioned earlier, our methodology consisted of both bulk log analysis and user interviews. We conducted semi-structured interviews, using the critical incident technique. How many of you are familiar with the critical incident technique? For those of you who aren't familiar with it, the critical incident technique is a methodology in which a subject is asked to reflect on a specific incident and discuss the cause, description and outcome of the incident as well as his/her feelings and perceptions of the situtation. Connaway, Lynn Silipigni, Chris Cyr, Brittany Brannon, Peggy Gallagher, and Erin Hood. 2019. “Speaking on the Record: Combining Interviews with Search Log Analysis in User Research.” ALISE/ProQuest Methodology Paper Competition Award Winner.    Go over methodology (semi-structured interviews with critical incident technique) Connaway, Lynn Silipigni, and Marie L. Radford. 2018. Survey Research. Webinar presented by ASIS&T, January 23. https://www.slideshare.net/LynnConnaway/survey-research-methods-with-lynn-silipigni-connaway and https://www.youtube.com/watch?v=4dlpAT7MXh0 .
• #13: Image https://www.flickr.com/photos/gpoo/293086443 by German Poo-Caamano CC BY 2.0
• #14: Image https://www.flickr.com/photos/enekobidegain/11327202173 by Eneko Bidegain CC BY 2.0 Peggy We developed specific protocols for each of our interviewees based on what we learned from their search logs. The protocols included some standard questions such as what they were looking for and why, whether they were successful – for example, if they were doing a known-item search, did the item appear in their search results, and whether they were delighted and/or frustrated with their search experience.
• #15: Image https://www.flickr.com/photos/eric_duquenoy/7244639090 by Eric Duquenoy CC BY-NC-ND 2.0 Peggy What we learned from her interview, however, was that she was just getting started searching for resources for a paper and hadn't yet narrowed down her topic. She was overwhelmed by how many search results she got, and so she did indeed abandon the library search. However, she came back to the library search tool at a later time, after she had done some Googling and honed in on her topic. She told us how she found the library search tool very helpful once she knew what she wanted to write about, and that she appreciated the help she received from some student workers. She also mentioned that because of the library instruction she had received in her first year, she felt confident in using the library.  CBU09 - Female, 19-25, undergraduate, humanities, Christian Brothers University
• #16: Peggy So, once we had conducted all of our interviews, we reviewed the interview transcripts, developed a code book and then had two different team members code each of the interviews. The themes that emerged from the interviews tended to fall into seven different buckets. Interviewees talked about the various search strategies they used and the factors that went into their decision-making process on which resources they wanted to access. As we've learned in several other studies over the years, convenience and ease of access are major factors. They talked about the formats they were most interested in – PDFs being the most mentioned. They talked about how they evaluate resources such as considering the type of study involved from a particular article and whether the article was peer-reviewed. They mentioned features they like about the current library search tool as well as features they wished it had. Many of them talked about being satisfied with their search experience but wouldn't go as far as saying they were delighted by it and spoke of some of their frustrations with the top frustration being too many search results. And many of them spoke favorably about the impact of working with a librarian for receiving library instruction on their confidence in their information-seeking abilities.  So, that's a quick review of what we did and how we did it. I'm going to turn things back over to Lynn who's going to talk about some of the challenges and benefits of the methodology we used for this study.
• #18: Image https://www.flickr.com/photos/eric_duquenoy/7244639090 by Eric Duquenoy CC BY-NC-ND 2.0 Brittany Once we had established those categories, we could begin looking at the overall patterns in our search logs. For the bulk log analysis, we looked at all of the logs in the Discovery system for April of 2018. We found that on average, a search session lasted 5 minutes and contained just over two searches, with an average of about 5 words per search. Twelve percent of sessions showed search refinements and 33% of sessions showed multiple searches.
• #19: Christopher Log events for last requests and for the full sessions were categorized in the following way: Online Access Attempt: The user clicked an item or made a request to digitally access the full text of the item.  Physical Access Attempt: The user clicked an item or made a request to place a hold on a physical copy of the item.  Physical Access Option: Some users left the system after looking at holdings. In these cases, users were in a place in the system where they were able to identify the physical item call number and/or location. The interviews revealed that at least some of the participants looked for the item on the library shelf at this point. Since logs cannot reveal what users did after leaving the system, these users were categorized as having the option to physically access the item.  Attempt to Save: The user made an attempt to export or otherwise save the citation.  Search Results: The user made a request for search results. This could include a new search, refinement of an existing search, or the addition of limiters.  Other: These requests and click events did not fit cleanly into any category, and there were no overarching themes to them.
• #20: Christopher While 54% of the total click events were to generate search results, only 39% of users left on a search results page. More than 50% of users left the system with a request to access an item online, an attempt to access an item physically, or an option to find its physical location. This discrepancy between all request categories and last request categories suggests that users were more likely to leave the system on a successful note. However, it cannot be determined whether the access attempts were successful, so conclusions must be made cautiously.  
• #21: Christopher
• #22: Image https://www.flickr.com/photos/boklm/37726606441 by Nicolas Vigier CC0 1.0 Lynn It’s changing the way we think, the way we work together. Research goes beyond the academic… influencing product development We earn the right to share our insights by listening.
• #24: Image:https://www.flickr.com/photos/satanoid/5851823198/by satanoid/CC BY 2.0
• #26: Image: https://www.flickr.com/photos/rexness/4513981469/ by Rexness/CC BY-SA 2.0
• #27: Image:https://www.flickr.com/photos/thetejon/4051985062/ by Jon/(CC BY-SA 2.0)
• #28: Image: https://www.flickr.com/photos/56001877@N04/8126838599/by runmonty/CC BY-NC-ND 2.0 Standardization: parsing multiple systems that don’t interact with one another. Makes fast fulfilment difficult for users. Request Volume: Librarians are frustrated by a high volume of short notice requests. Usually from students working on assignments. Funding: Librarians operate on limited funds to meet user fulfilment requests
• #29: Image: https://www.flickr.com/photos/strausser/10926019275/ by free range jace/CC BY-NC 2.0 Libraries do business with multiple ILL consortia that have high variation between their systems that makes both the logistics of shipping items and giving projected transit time difficult.
• #30: Image: https://www.flickr.com/photos/loloieg/3459011184/ by Loloieg/CC BY-ND 2.0 Librarians are held responsible by users when something goes wrong at any point in the ILL process even though the shipping phase can be prone to errors and slowdown outside of librarian control
• #31: Image: https://www.flickr.com/photos/hsin-yen/35173724402/by Hsin-Yen Lin/CC BY-NC-ND 2.0 Many librarians want to expand the resources offered through ILL but funding is a limiting factor.
• #33: Image: https://www.flickr.com/photos/thaths/4314445838/ by thaths/(CC BY-NC 2.0)
• #34: Image: https://www.flickr.com/photos/skellner/4203154536/ by Stefan Kellner/CC BY-NC 2.0
• #35: Image: https://www.flickr.com/photos/duncan/15262071481/by Duncan c/CC BY-NC 2.0
• #36: Image:https://www.flickr.com/photos/fransdewit/4052612802/by Frans de Wit/CC BY-NC-ND 2.0 Speed stood out as the most important aspect of ILL even overshadowing quality. ILL is frequently seen as not worth the time needed and unless the item is mandatory it will be skipped by the user. Digital items are preferred when high speed is a necessity.
• #37: Image: https://www.flickr.com/photos/oprior/64409980/by Owen Prior/CC BY-SA 2.0
• #38: Image:https://www.flickr.com/photos/jhlau/15969832976/ by a.canvas.of.light/CC BY 2.0
• #39: Image: https://www.flickr.com/photos/davem/14803922/ by Dave & Lorelle/CC BY-NC 2.0 While price points weren’t mentioned as specifically high or low the issue of paid access vs free access was frequently mentioned. Paid options were unanimously ignored in the presence of a free alternative. If the first choice item turns out to be paid only the user will pass it up in favor of a different item. Users broadly wanted more items to be available for free even outside of universities.
• #40: Image: https://www.flickr.com/photos/101643643@N08/15749180169/ by WANG-HSIN PEI/CC BY 2.0
• #41: Image:https://www.flickr.com/photos/laurie_brooker/354514974/by Laurie Brooker/CC BY-NC-ND 2.0 Item format (digital physical) was dependent on the situation and the needs of the user except for a few “format loyalists” who overwhelmingly preferred one format type to another.
• #42: Image: https://www.flickr.com/photos/130540439@N05/16544443591/by LT Photography/CC BY-NC-ND 2.0 Journals and eBooks were the most frequently mentioned digital resources, accessed through google scholar, specialized databases, and amazon. Physical journals were never mentioned with the exception of remarking that the user had never worked with them or worked with them a long time ago but not anymore. Speed, flexibility and portability were commonly mentioned as the best aspects of digital media.
• #43: Image:https://www.flickr.com/photos/kool_skatkat/795393193/by kool_skatkat/(CC BY-NC-ND 2.0)
• #44: Image: https://www.flickr.com/photos/ncole458/5251071720/ by Nicholas Cole/CC BY 2.0 Books were overwhelmingly the most popular physical media with immersion into the text (sitting down and reading for an extended period) and the ability to pick up and read the book offline were frequently mentioned. Additionally, books were mentioned for both use in personal and professional settings. While the individual materials didn’t often overlap, the medium of print books was still ubiquitous.
• #45: Image:https://www.flickr.com/photos/thetejon/4182681980/by Jon/(CC BY-SA 2.0)
• #46: Image: https://www.flickr.com/photos/huangce/8390725539/ by TripNotice.com/CC BY 2.0 Many users have a connotation that ILL is slow and unreliable, this is reinforced by their own avoidance of ILL (self-fulfilling prophecy). Those who do use ILL tend to be satisfied with it but see where improvements could be made. The issue of clarity is important. Many users complain about anxiety in not knowing when their item will arrive or if it will arrive on time. In contrast to this users also mentioned not wanting to be swamped with updates across multiple channels. Emails at key points in shipping were most often cited as ideal.
• #47: Image: https://www.flickr.com/photos/rogermeyer/5244084417/ by Rodger Meyer/CC BY-NC-ND 2.0
• #48: Image:https://www.flickr.com/photos/fernando/3992060782/by Fernando de Sousa/CC BY-SA 2.0
• #49: Image: https://www.flickr.com/photos/147621582@N02/35993391071/by Staffan Andersson/CC0 1.0 Open access was occasionally mentioned but for a handful of interviews it was a key issue they spoke about at length. Open access wasn’t commonly talked about as a present issue but more so as a future goal. Something that would be nice to pursue eventually, once other issues were addressed.
• #50: Image: https://www.flickr.com/photos/karlnorling/14668687253/by karlnorling/CC BY 2.0
• #51: Image: https://www.flickr.com/photos/duncan/39921909493/by Duncan c/CC BY-NC 2.0
• #52: While both groups cared about convenience and speed, graduate/postgraduate students appear to be more sensitive to it. Because of this, they have a much bigger preference for PDFs than faculty. Faculty are more interested in open content than graduate/postgraduate students While Americans and Australians differed in their wish for clarity in communication, faculty and graduate/postgraduate students do not appear to differ at all. Image: https://www.flickr.com/photos/shiridenovo/5318874705/ by Carey Ciuro/CC BY-NC 2.0
• #53: Image:https://www.flickr.com/photos/majiksbox/24923950387/by Dave – resting/CC BY-SA 2.0
• #54: Image https://www.flickr.com/photos/axelrd/4019174359 by Axel Drainville CC BY-NC 2.0