Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage
1 like1,341 views
The document describes a large-scale study analyzing the effect of discovery systems on online journal usage across multiple libraries and publishers. Statistical analyses were conducted to determine if observed differences in journal usage before and after implementing discovery services were significant. The analyses found that while the impact varied, discovery service, publisher, and library were all significant predictors of usage change. Specifically, the interaction between discovery service and publisher significantly impacted usage. The study suggests analyzing journal usage is complex with many factors requiring control.
![Prior
studies
• Some
studies
have
indicated
substanJal
increases
in
usage
a]er
Discovery
implementaJon
– DescripJve
staJsJcs
only
– Single
insJtuJon
studies
only
• Some
publishers
report
decreased
usage
of
content
– Anecdotal,
may
affect
some
and
not
others](https://image.slidesharecdn.com/mlcjdmjspcharleston2013slideshare-131111223638-phpapp02/85/Discovery-or-Displacement-A-Large-Scale-Longitudinal-Study-of-the-Effect-of-Discovery-Systems-on-Online-Journal-Usage-9-320.jpg)
![Methodology
Compared
COUNTER
JR1
total
full
text
arJcle
views
for
the
Included
implementaJon
month
in
Year
1
to
ensure
that
both
periods
included
an
enJre
academic
year
End
Year
2
May
2012
Year
1
ImplementaJon
May
2011
Start
June
2010
12
months
before
vs
12
months
a]er
implementaJon
date](https://image.slidesharecdn.com/mlcjdmjspcharleston2013slideshare-131111223638-phpapp02/85/Discovery-or-Displacement-A-Large-Scale-Longitudinal-Study-of-the-Effect-of-Discovery-Systems-on-Online-Journal-Usage-14-320.jpg)
Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage
- 1. Discovery or Displacement? A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage Charleston Conference November 7, 2013 Michael Levine-‐Clark, University of Denver John McDonald, University of Southern California Jason Price, SCELC ConsorJum
- 2. “…a steep increase in full text downloads and link resolver click‐ throughs suggests Summon had a dramatic impact on user behavior and the use of library collections during this time period.” The Impact of Web-scale Discovery on the Use of a Library Collection Doug Way (2010) http://scholarworks.gvsu.edu/ library_sp/9/
- 6. Web-‐scale discovery services • Single source for finding informaJon – Books – ArJcles – Local content • Metadata and/or full text • Content is pre-‐indexed and/or pre-‐harvested • Single fast search ILS Publisher Metadata MLA Bibliography InsJtuJonal Repository HathiTrust Discovery Service
- 7. An assump5on • At any given insJtuJon, given a relaJvely stable user base, the total search effort will remain roughly the same. – X students will have Y assignments and Z hours per day to search – X faculty will publish Y papers and have Z hours per day to search
- 8. Discovery services § Will take up an increasing amount of a finite Jme for searching § Will draw users from other (more or less efficient) search tools § Will alter the overall producJvity of searches (users will find more or less) § Will alter the overall efficiency of users (users will access more or less full-‐text)
- 9. Prior studies • Some studies have indicated substanJal increases in usage a]er Discovery implementaJon – DescripJve staJsJcs only – Single insJtuJon studies only • Some publishers report decreased usage of content – Anecdotal, may affect some and not others
- 10. Data collec5on • List of libraries with discovery services > Searched on lib-‐web-‐cats • Surveyed Libraries > Discovery service Implemented > ImplementaJon Date (month/year) > Search box locaJon > MarkeJng effort • 149 Libraries Gave Approval > 24 libraries selected for this phase > 6 for each of the 4 major discovery services
- 11. Library demographics • 20 US, 1 each from UK, AUS, NZ, CA • 10 ARL Libraries included • WorldCat book holdings > Average: 1,114,193 > Median: 1,044,153 > High: 2,665,796 > Low: 298,365 • ImplementaJon dates: > 2010 (3), 2011 (19), 2012 (2)
- 13. Dataset • 24 Libraries • 4 Discovery services • 6 Publishers • 9,206 Journals • 159,278 ObservaJons • 141,048 Usable ObservaJons
- 14. Methodology Compared COUNTER JR1 total full text arJcle views for the Included implementaJon month in Year 1 to ensure that both periods included an enJre academic year End Year 2 May 2012 Year 1 ImplementaJon May 2011 Start June 2010 12 months before vs 12 months a]er implementaJon date
- 15. Collec5ons notes o Excluded journals that did not have 24 months of COUNTER reporJng o Limited ability to control for changes in aggregator, backfile access, or expanded holdings o Outliers removed from analysis
- 22. General trends • VariaJon by insJtuJon within each discovery service • VariaJon by publisher within each discovery service • Some publishers saw overall net increase, while some experienced a decrease in usage
- 24. Goals of our inferential statistics Determine whether observed differences are significant or resulted from chance effects Determine which of the three factors (i.e. library, publisher, discovery service) contribute to determining differences in usage change at the journal level Start with an exploratory analysis and end with a comprehensive model
- 25. ANOVA - Analyzing the data Observation = Fit + Residual Change In = Library x + Publisher y + Disc Svc z + Residual Err usage +17 = (+2) + (-3) + (+10) + (+8) After Cobb 2003 Introduction to design and analysis of experiments. Fig 3.1
- 26. ANOVA – F Ratio Tests whether the means for levels within a factor are distinguishable from each other Average variability due to the factor F-ratio = --------------------------------------------------Average variability due to chance error So, when F ≈ 1, means are not distinguishable, when F is > 1, there are real differences among some means
- 27. Does usage change vary across libraries? Overall Average = 8.5 Institution (sorted by Mean Change)
- 28. Does usage change vary across libraries? Overall Average = 8.5 Institution (sorted by Mean Change)
- 29. Does usage change vary across publishers? Overall Average = 8.9 Publisher (sorted by Mean Change)
- 30. Does usage change vary across discovery services? Overall Average = 8.9
- 31. Does the affect of discovery service differ across publishers?
- 32. Does the affect of discovery service differ across publishers? Publishers (distinguished by color)
- 33. Do the discovery service means differ in the 2 way model? 15.0 12.3 4.5 3.7 Publishers (distinguished by color)
- 35. Do publisher means differ significantly in the two way model? 23.8 6.8-9.5 -3.9 Publishers (distinguished by color)
- 36. Do publisher means differ significantly in the two way model? 23.8 6.8-9.5 -3.9 Publishers (distinguished by color)
- 37. Does the affect of discovery service differ across publishers? Publishers (distinguished by color)
- 38. Publishers (distinguished by color)
- 39. Full Model – including disc srvc, publisher, and library
- 40. ANOVA Model including all three factors
- 41. Results - Can we detect differences between Discovery Services, Publishers, and/or Libraries and/or their interactions? Discovery Service – Yes Publisher – No Library – Yes Differential discovery service effect by publisher – Yes Differential library effect by publisher -- Yes
- 42. Interpreta5ons & Conclusions > > > > > Analyzing usage is a complex task No discovery service increased or decreased usage across all libraries and/or all publishers Discovery service and publisher as variables on their own were significant predictors of usage change InteracJon of Discovery service & Publisher was significant Some control needed for no discovery service and for size of insJtuJon.
- 43. A plethora of pending possible pursuits • Design & test for effects of: – – – – – – Aggregator full text availability InsJtuJon Size / Enrollment Profile Publisher Size Journal Subject Overall usage trends (Requires Disc Srvc ‘control’) ConfiguraJon opJons in Discovery services • Follow-‐up presenta5on at UKSG (April 2014) – Including Control group & AddiJonal libraries – Add AddiJonal variables & further analysis
- 44. Ques5ons michael.levine-‐clark@du.edu | johndmcd@usc.edu | jason@scelc.org