Practical applications of citation analysis-handout
Download as DOCX, PDF1 like1,224 views
The document discusses practical applications of do-it-yourself citation analysis, emphasizing its role in measuring the impact of journals and authors objectively. It provides a basic method for conducting citation analysis, highlights important tools like WorldCat and PsycInfo, and underscores the complexities of interpreting citation data due to factors like data errors and the Matthew effect. Additionally, it includes an annotated bibliography and critiques on the reliability of impact factors in research evaluations.
Practical applications of citation analysis-handout
- 1. Practical Applications of Do-It-Yourself Citation Analysis Steve Black Serials and Reference Librarian The College of Saint Rose, Albany, NY June 11, 2012 Rationale for using citation analysis: The study of patterns and frequencies of citations is an objective, quantitative way to measure the impact of journals, authors, institutions, nations, etc. For librarians, citation analysis can help one identify journals for supporting research, building collections, or submitting papers for publication. Impact factor (simple version): cites to articles published in last 2 years ÷ number of articles published in last 2 years Full definition at http://thomsonreuters.com/products_services/science/free/essays/impact_factor/ Critiques of relying on impact factors include: (See bibliography) Data errors, both in the citations themselves and in how they are compiled by the citation indexes Matthew effect, coined by Robert Merton after the passage in Matthew (13:12), ―for unto every one that hath shall be given, and he shall have abundance: but from him that hath not shall be taken away even that which he hath.‖ The more attention a work gets, the more attention it gets, skewing citations to papers that are notable for being noted, not necessarily because they’re the highest quality or the most important. Impact vs. quality vs. importance. Impact is how often cited. Quality is the caliber of the work—elegance of method, thoroughness of literature review, quality of writing, etc. Importance is contribution to a line of inquiry, regardless of how narrow. For various reasons high quality and very important papers may not be cited much. Global vs. local impact. Citation in publications may poorly correlate with the needs of local patrons. 1
- 2. Basic DIY Citation Analysis Method • Select a target population, e.g. – journals in a sub-discipline – group of researchers – subject(s) or keyword(s) • Select a sample that represents the target population • Compile works cited in the chosen sample • Sort and count the works cited Case study based on author’s ―Frequently Cited Journals in Forensic Psychology,‖ Psychological Reports, v.110, no.1, (2012): 276-282. Tools used: • WorldCat (to identify most widely held journals) • PsycINFO via EBSCOhost (to identify works cited) • RefWorks (to organize works cited) • Excel (to count and rank cited journals) Notes on these tools: Our ability to use WorldCat holdings as a measure of importance may fade over time as fewer libraries subscribe to individual titles that appear as holdings in OCLC PsycINFO is particularly well suited to the task because all works cited are included in the database whether the article is in full text or not, and are in a format that is easily exported. If you do a similar project, seek out data sources that allow easy downloading of works cited. RefWorks is only one of many citation management tools that can be used for organizing citations. Excel has a COUNTIF function that could be used to count times cited, but it will undercount if there are any variations in spelling, punctuation, or spacing. It’s more accurate to count by highlighting the titles and noting the count of highlighted rows. 2
- 3. Ranks of cited journals by single issues of Law and Human Behavior Title cited in LHB 2011 frequency all Allrank FebRank AprilRank JunRank AugRank OctRank DecRank Law and Human Behavior 154 1 1 1 1 1 1 1 Criminal Justice and Behavior 45 2.5 9.5 6 2 2 9.5 Journal of Applied Psychology 45 2.5 2 4.5 3 4.5 4 Behavioral Sciences and the Law 40 4 3 6 4 2 16 Journal of Personality and Social Psychology 35 5 16.5 7.5 3 8.5 9.5 11.5 Psychological Assessment 32 6 3 13 8.5 11 4 Applied Cognitive Psychology 29 7 4 17.5 13 12 13.5 2 American Psychologist 26 8.5 16.5 9.5 8.5 9.5 8.5 Psychiatric Services 26 8.5 2 16.5 6 Journal of Abnormal Psychology 21 10.5 5 13 9.5 11.5 Psychology, Public Policy, and Law 21 10.5 4.5 4.5 16 Journal of Consulting and Clinical Psychology 18 12 8.5 5.5 21 Personality and Individual Differences 15 13.5 4 16 Psychological Bulletin 15 13.5 11.5 13 8.5 Assessment 14 15.5 6.5 8.5 21 Psychological Review 14 15.5 3 21 Journal of Applied Social Psychology 13 18 16.5 17.5 10 Journal of Experimental Psychology: Applied 13 18 16.5 8.5 Psychological Science 13 18 16.5 11.5 Psychology, Crime and Law 12 20 4 International journal of law and psychiatry 11 21.5 16.5 13.5 Legal and Criminological Psychology 11 21.5 17.5 13 16 Tips on assigning ranks • In order to test reliability, the number of ranked items in each group must be equal. In this case December had only 22 titles with 2 or more times cited, so the top 22 ranked journals in each group were compared. • Handle ties by averaging, e.g. tied for 2 & 3 are each ranked 2.5. • Assign bottom rank to titles with no match, e.g. if n=22, unmatched titles are ranked 22. Formula for Spearman’s rho rank correlation: NOTE: This is the simple version of rsthat assumes no ties in ranks. See bibliography under ―Technical‖ for sources of the more complex formula. In practice the two formulas return very similar resultseven with some ties between rankings. 3
- 4. rs correlations among issues of LHB 2011 Feb Apr June Aug Oct Dec 2011 1 Feb 0.24 1 Apr 0.47 -0.34 1 June 0.26 -0.05 -0.20 1 Aug 0.63 0.05 0.40 0.15 1 Oct 0.59 -0.05 0.46 -0.47 0.11 1 Dec 0.23 0.25 -0.66 -0.42 -0.03 -0.02 1 Weak, scattered correlations indicate very low reliability of ranks at the single issue level, which for this case n=250-350. rs correlation of LHB (2011) vs. ranking published in Psychological Reports LHB Psych 2011 Reports D D2 title rank rank Law and Human Behavior 1 1 0 0 Criminal Justice and Behavior 2.5 3 -0.5 0.25 Journal of Applied Psychology 2.5 5 -2.5 6.25 Behavioral Sciences and the Law 4 2 2 4 Journal of Personality and Social Psychology 5 6 -1 1 Psychological Assessment 6 12 -6 36 Applied Cognitive Psychology 7 9 -2 4 American Psychologist 8.5 14 -5.5 30.25 Psychiatric Services 8.5 20 -11.5 132.25 Journal of Abnormal Psychology 10.5 22 -11.5 132.25 Psychology, Public Policy, and Law 10.5 10 0.5 0.25 Journal of Consulting and Clinical Psychology 12 16 -4 16 Personality and Individual Differences 13.5 13 0.5 0.25 Psychological Bulletin 13.5 22 -8.5 72.25 Assessment 15.5 22 -6.5 42.25 Psychological Review 15.5 22 -6.5 42.25 Journal of Applied Social Psychology 18 21 -3 9 Journal of Experimental Psychology: Applied 18 22 -4 16 Psychological Science 18 22 -4 16 Psychology, Crime and Law 20 11 9 81 International journal of law and psychiatry 21.5 22 -0.5 0.25 Legal and Criminological Psychology 21.5 15 6.5 42.25 sum of D squared 684 rs=1-(6*684)/22*(222-1) = 1 – (4104/10626) = 1- .37 rs= .63 4
- 5. ROUGH rule of thumb for reliability of ranking by sample size Sample approximate expected correlation with population sample size (works cited)* one issue of journal n=250 very low one volume of journal n=1500 moderate several volumes of 3-5 journals n>10,000 strong *Of course the number of works cited in a volume or issue will vary depending on the journal Measure variation of individual titles’ frequency of citation with Coefficient of Variation = standard deviation ÷ mean Times Cited in 2011 Law and Human Behavior Coefficient of Feb Apr June Aug Oct Dec σX mean Variation Law and Human 28 30 17 25 27 27 4.55 25.67 18% Behavior Journal of Applied 13 9 0 8 8 7 4.23 7.50 56% Psychology Psychology, Public 1 9 0 0 8 3 4.04 3.50 115% Policy, and Law In general, titles further down ranked lists have higher coefficients of variation. A higher coefficient of variation means less reliability of the ranking— it’s more likely that a different sample will result in a much different rank. 5
- 6. Statistical analysis of this case suggests that so long as the sampled journals validly represent the topic,a sample of • n<1,000 works cited can indicate the top journal (if any) • n>1,000 can generate a rough indication of leading journals • n>10,000 can create a useful ranked list CAUTIONARY NOTE Even with n>10,000, different samples will yield different ranks, especially further down the list, so a journal’s rank must ALWAYS be taken as an approximation of its ―true‖ ranking. Since rankings reflect the complex reasons researchers cite one thing or another, there can never be a static, definitive ranking for journals in any topic. Similar statistical analysis I’ve done onjournals in communication disorders suggests that a third of movement in rankings over time is due to random variation. Tips for publication • Don’t be too parochial or narrow—editors must think their readers will be interested. • BUT choose something not already in Journal Citation Reports. • Be very thorough with your lit review before gathering data (search disciplinary databases as well as the library literature). • Gather citations and test sample before nailing down your method. • Take the time and energy to ensure sample is o representative o clearly defined o robust o valid on its face • Group data and run statistics by journal volume • Consider submitting to a journal in the topic area 6
- 7. PRACTICAL APPLICATIONS OF DIY CITATION ANALYSIS What’s the top journal on this topic? – Compile cites from 10-20 articles on the topic to identify it (or find there’s not one). What does my library need to support a proposed new major? – Compile cites from 1-2 volumes of 2-4 journals to flag titles not already available. Which journals are REALLY used by that herd of cats in Dept. X? – Analyze publications & dissertations from last decade. OMG I’m up for tenure soon! – Identify a topic and do a thorough analysis of 2-3 volumes of 4-6 journals; include reliability tests. 7
- 8. CITATION ANALYSIS: A SELECTIVE ANNOTATED BIBLIOGRAPHY CLASSICS S. C. Bradford, “Sources of Information on Specific Subjects,” Engineering 137 (1934): 85-86. This study of the concentration of citations to literature in applied geophysics and lubrication is the original source of Bradford’s Law of Distribution. Eugene Garfield, “Citation Analysis as a Tool in Journal Evaluation,” Science 178 (1972): 471- 479. Garfield explains and argues for the Institute for Scientific Information’s Science Citation Index. Robert K. Merton, The Sociology of Science: Theoretical and Empirical Investigations (Chicago: University of Chicago Press, 1973). Includes the famous sociologist’s description of how the Matthew Effect impacts scientific output. OVERVIEWS Linda C. Smith, “Citation analysis,” Library Trends 30 (1981): 83-106. One of several important articles in a special issue of Library Trends devoted to bibliometricsin the early years of the “serials crisis,” when many libraries were first seriously confronted with having to decide which journals to cut. Thomas E. Nisonger, “Chapter 5, The Application of Citation Analysis to Serials Collection Management,” Management of Serials in Libraries (Englewood, CO: Libraries Unlimited, 1998): 121-156. A well-organized presentation of major issues and a thorough bibliography. Thomas E. Nisonger, “Journals in the Core Collection: Definition, Identification, and Applications,” Serials Librarian 51, no. 3/4 (2007): 51-73. Summarizes ten methods for creating lists of core journals and discusses applications of core lists. CRITICAL ANALYSES Michael H. MacRoberts and Barbara R. MacRoberts, "Problems of Citation Analysis: A Critical Review," Journal of the American Society for Information Science 40, no. 5 (1989): 342-349. A thorough and well organized critique with specific emphasis on Science Citation Index. Per O. Seglen, “Why the impact factor of journals should not be used for evaluating research,” British Medical Journal 314 (1997): 498-502. Concise summary of the problems associated with the use of journal impact factors. 8
- 9. Maurice B. Line, "Changes in Rank Lists of Serials Over Time: Interlending versus Citation Data," College & Research Libraries 46, no. 1 (1985): 77-79. Robert N. Broadus, "A Proposed Method for Eliminating Titles from Periodical Subscription Lists," College & Research Libraries 46, no. 1 (1985): 30-35. Maurice B. Line, "Use of Citation Data for Periodicals Control in Libraries: A Response to Broadus," College & Research Libraries 46, no. 1 (1985): 36-37. These 3 articles comprise a dialogue between Broadus and Line on the validity of citation data as a tool for collection development in an era of cost-cutting. R.E. Rice, Christine L. Borgman, Diane Bednarski, and P.J. Hart, “Journal-to-Journal Citation Data: Issues of Validity and Reliability,” Scientometrics 15, no. 3 (1989): 257-282. Reviews issues of validity and reliability, discusses causes of measurement errors, and concludes with suggestions for how to reduce measurement errors. Ben R. Martin, "The Use of Multiple Indicators in the Assessment of Basic Research," Scientometrics 36, no. 3 (1996): 343-362. Defines quality, importance, and impact and stresses the importance of respecting each. Gordon and Breach Science v. American Institute of Physics and American Physical Society, http://barschall.stanford.edu. This web site contains a thorough and well organized treatment of the Gordon and Breach case against Henry H. Barschall and the publishers of his studies. Barschall used citation counts and subscription costs to create rankings of physics journals. Gordon & Breach titles were shown to be among the poorest values in physics, and they sued the publisher for false advertising. The court records and related documents do an excellent job of presenting the issues surrounding applications of citation analysis. METHODS Steve Black, “Using Citation Analysis to Pursue a Core Collection of Journals for Communication Disorders,” Library Resources & Technical Services 45, no. 1 (2001): 3-9. Includes a basic method for do-it-yourself citation analysis. Steve Black, “How Much do Core Journals Change over a Decade?” Library Resources & Technical Services 56, no.2 (2012): 80-93. Describes methods for correlating ranked lists over time. Jeffery D. Kushkowski, Kristin H. Gerhard and Cynthia Dobson, “A Method for Building Core Journal Lists in Interdisciplinary Subject Areas,” Journal of Documentation 54, no. 4 (1998): 477-88. Describes a Simple Index Method for ranking journals based on results of subject or keyword searches in relevant databases. 9
- 10. Daniela Rosenstreich and Ben Wooliscroft, “Measuring the impact of accounting journals using Google Scholar and the g-index,” British Accounting Review 41 (2009): 227-239. Valuable for its treatment of Google Scholar, a table summarizing common criticisms of citation-based journal rankings, and comparisons of ranking methods. (Also a good example of how an important paper can be published in an unexpected place!) Chris Piotrowski, “Top cited journals in forensic psychology: An analysis of the psychological literature, “American Journal of Forensic Psychology 30, no. 2 (2012): 29-37. An example of using keyword searches to rank journals. Piotrowski’s method yields a very different ranked list from my list published in Psychological Reports. TECHNICAL Thomson Reuters, “The Thomson Reuters Impact Factor,” http://thomsonreuters.com/products_services/science/free/essays/impact_factor. An overview by Eugene Garfield that includes the formula for calculating impact factor, the rationale for using it, caveats and cautions. Stephen J. Bensman, "Probability Distributions in Library and Information Science: A Historical and Practitioner Viewpoint,” Journal of the American Society for Information Science and Technology 51, no. 9 (2000): 816-833. Argues that parametric statistics based on Poisson distribution are incapable of accurately modeling patterns of journal citations. Sidney Siegel, "Nonparametric Statistics," American Statistician 11, no. 3 (1957): 13-19. An authoritative, readable description of when and why to use various statistical methods including Spearman’s rho. Maurice Kendall and Jean Gibbons, Rank Correlation Methods. 5th ed. (New York: Oxford University Press, 1990). Complete descriptions, formulas, and proofs of Spearman’s rho and Kendall’s tau (a calculation based simply on whether items go up or down in rank, disregarding the degree of change). Steve Black blacks@strose.edu (518) 458-5494 10