Using an Availability Study to Assess Access to Electronic Articles

Editor's Notes

• #2: Introduce self. Note: This work was done at my prior library, Oregon Health & Science University, with help from my colleague, Carla Pealer.
• #3: Including this information for context but will go through it quickly in the interest of time.
• #4: Focus of today’s presentation will be methodology rather than results, though I will summarize our major findings [Note: cut that part if time is short]. Want to share methodology that you can apply in your own setting. Detailed results can be found in the conference paper.
• #5: Like most libraries, OHSU uses a suite of tools to manage electronic resources and provide access to them. Most of these were from Innovative Interfaces, except for the proxy server. Again, just context for our findings.
• #6: Now that we’ve covered the context, let’s talk about the problem we were trying to solve. We had lots of data related to use of our electronic collections, but none of them measured the complete experience of retrieving an article. Usage data – quantity only. Can’t see what users want but don’t get, either b/c we don’t have it, or something isn’t working. Also can’t see if they encounter problems trying to access an article.Support requests provide anecdotal data on problems. But hard to know whether a problem is widespread, how much time and energy we should spend trying to solve it. Also, users may not submit requests because they eventually found a way to get the article—or because they gave up. Need more solid data to make decisions about collections and access.Usability tests are excellent for identifying problems with web interfaces, but they involve a short list of tasks created by library staff rather than actual user requests. So they don’t reflect user demand, and they’re unlikely to include items for which there are problems. They are designed to identify problems with user interfaces rather than with holdings or retrieval tools.We didn’t know how successful users are at getting the articles they want. We also didn’t know all the factors that get in their way, and how often these problems occur. Needed another way to measure the quality of our electronic collections and access to them.
• #7: In a nutshell, when you do an availability study, you gather actual user requests, try to fill them, and see what happens. You record what happens, noting the number and nature of any problems you encounter, and analyze the data to learn: How often your library is able to satisfy a user’s request for an item (in our case, a journal article)What barriers users encounter when trying to retrieve itemsMethod has been around awhile; first described by Paul Kantor in 1976.
• #8: And now for the world’s shortest literature review. If you want to do an availability study, I recommend the two articles by Nisonger. That’s where I started, and they were very helpful. Full citation is in the conference paper.There have been very few published availability studies involving electronic articles, and I couldn’t find any that involved a link resolver, as this one did.
• #9: So, now that we’ve covered the background, let’s take a look at what we did. This slide summarizes the steps. We’ll look at each step in more detail. Basically, we used link resolver log data to get a sample of articles users had tried to retrieve via the link resolver. We used the resolver and the library catalog to try to retrieve the articles, recorded the results of our testing plus some basic info about the article, and analyzed the results.
• #10: Innovative, our link resolver vendor, was able to send us some log files, which included each openURL received and processed by the resolver for a given period of time, as well as which options in the link resolver menu the user clicked. In other words, the logs told us which articles users tried to access – data we’d never had before. Our log files covered parts of days during 2 three-week periods last fall and this spring. We had to clean the data a bit, removing entries for web page elements like images and CSS. Then we were ready to test a sample.
• #11: We tested every 3rd entry, skipping duplicates (where user clicked the same item more than once in rapid succession) and entries for anything that wasn’t a journal article.
• #12: For testing, we tried to replicate what a user would do to retrieve full text using our two primary tools, the link resolver and the catalog. For link resolver, we pasted the openURL from the log into a web browser. That took us to a menu of options in our link resolver (as shown in the screenshot on this slide), from which we attempted to retrieve full text.
• #13: After we attempted to retrieve the article via the link resolver, we tried to retrieve it via the catalog, which includes all the OHSU Library’s journal holdings, print and electronic. This screenshot shows part of a journal record display in the OHSU Library catalog—the part that shows electronic holdings.
• #14: As we tested, we recorded our results, documenting whether or not the article could be retrieved by each method and the nature of any problems encountered. When using the catalog, if an article was not available electronically, but the catalog indicated that it was available in print, we noted that. We didn’t go to the shelf to verify that it was really there.
• #15: In addition to documenting the results of our testing, we recorded basic information about each article tested, so we could see if/how article characteristics correlated with availability. We noted the database from which the link resolver request originated and whether or not the user clicked any links in the link resolver menu. We also noted the journal title and publication year of the article.
• #16: We analyzed results to determine the percentage of articles that were available electronically or in print, breaking down the results by publication date and origin. We also analyzed the nature and frequency of problems we encountered.Now I’ll show you a few selected results to give you a sense of what we did. Complete results are in the paper, but these examples will give you a sense of the kind of information you can glean from an availability study. I’m skipping over most of the results, because while the results are interesting to OHSU, results at other libraries are likely to be quite different. The methodology is what is likely to be useful to you.
• #17: This chart shows availability via the link resolver and catalog by format. Note especially that 27 items were available electronically via the catalog but could not be retrieved with the link resolver. If you’re interested, results are discussed in depth in the paper.
• #18: Here’s a list of the problems encountered when trying to retrieve electronic articles via the catalog. Lack of holdings was the biggest problem, accounting for 3 of the top 4 problems. While that result isn’t surprising, it could be useful to have the problem broken out into components—no holdings at all vs. print only, etc.—and the study provides data on which to base collection and resource allocation decisions.Note that I’ve arranged the problems in order from most to least common. One of the most useful things about an availability study is that it can help you allocate/prioritize resources (staff and money) to fix the biggest problems. The next slide shows another way to look at this data to illustrate where you will get the most value for your time and money.
• #19: A Pareto chart shows graphically which problems are most common, arranging problems in order from most to least common and showing the cumulative percentage as you move from most to least common. They are useful for showing where to devote resources to get the most results. The line across the top shows the cumulative percent of problems. So, for example, the first three categories represent 81% of the problems encountered.
• #20: The other major category of results came from trying to retrieve articles via our link resolver. This chart shows the problems encountered. Note that this list includes only problems specific to the link resolver. If we had no holdings, we weren’t able to retrieve the item via the link resolver either. Also note that in some cases, there was more than one problems with a single article.Interestingly, over half of the problems could be traced back to the metadata transmitted from the origin database, through the link resolver, to the full text provider. There are lots of possible points of failure in that chain. Many of those points of failure are out of the library’s control, but findings like this can help us lobby for higher-quality metadata to improve access for patrons.
• #21: And here’s the Pareto chart for the link resolver problems. Rest of findings are in the paper. As I mentioned, I wanted to spend presentation time on methodology, given the audience. Plus, my findings are unique to one institution, but the methodology can be useful nearly anywhere. Since there are so many, it was hard to fit on a ppt slide and still keep it readable.
• #22: Learned a lot about what gets in the way of users trying to access full text, which should make it easier to focus resources in areas that will make the most difference to users. Have data rather than just anecdotal information. Tells a lot about how well our collections—and the tools to access them—meet the needs of our users. Also, discovered that link resolver logs are a rich source of information about what users are seeking. Don’t know if other link resolver products generate logs or if it’s possible to get access to them, but if so, I highly recommend it. In addition to an availability study, you could mine the logs to see which journals are most in demand—especially which ones the library does not own. Could also identify which materials owned in print only are high demand and therefore a top priority to get electronically.Also provides a baseline. Could re-evaluate after making changes based on results, be able to show that the changes led to improvements for users. Finally, WebBridge doesn’t have logs that are accessible to the library. Only got access because we asked the product manager, who arranged for us to get the data.
• #23: I’d be glad to take questions now, or feel free to contact me after the conference. I’d love to hear from you.