Citation & altmetrics - a comparison
5 likes2,455 views
The document outlines various metrics used to evaluate scholarly products, including Journal Impact Factor, Eigenfactor Score, H-index, and Article Influence Score, detailing how they are calculated and their applications. It emphasizes the importance of both traditional metrics and emerging altmetrics for measuring the impact of research outputs. Additionally, the document discusses the challenges and considerations in using these metrics, such as data availability and variability across different sources.
Citation & altmetrics - a comparison
- 1. IUPUI University Library Center for Digital Scholarship: Metrics for Scholarly Products Created by Kellie Kaneshiro & Heather Coates, rev. 5/2015 1 Metric How Calculated Update Frequency Source Keep in mind Use it For Journal Impact Factor (JIF) Calculated annually from average number of citations received per paper during the 2 preceding years Calculation is based only on journals indexed by Thomson Reuters (citation-based) Full year’s data necessary before calculating 2011-2012 data will not be ready until summer 2013 Proprietary algorithm Published in database: Journal Citation Reports (JCR) from Thomson Reuters (ISI) http://thomsonreuters.com/products_services/sci ence/science_products/a- z/journal_citation_reports/ • Journal level metric • Calculated only for JCR Journals • Journal self-citations included in calculation • May be influenced by editorial policies • Targeting journals in which to publish • Identifying journals relevant to a specific discipline • Measuring a journal’s status Eigenfactor Score Based on the number of times articles from the journal published in the past five years have been cited in the JCR year and takes into account which (highly cited or less highly cited) journals have contributed to these citations Journal self-citations are removed (citation-based) Updated with each new release of JCR Impact Factors Algorithms and methodology are described at the website: http://www.eigenfactor.org Published in Journal Citation Reports, and at Eigenfactor website • Journal level metric • A journal’s Eigenfactor score doubles when it doubles in size – the more articles a journal publishes, the higher the Eigenfactor • Targeting journals in which to publish • Identifying journals relevant to a specific discipline • Measuring a journal’s status Article Influence Score Calculated from the journal’s Eigenfactor Score divided by the normalized fraction of all articles published in all journals. The mean score is 1.00, greater than 1.00 indicates above average influence, and less than one below average influence. (citation-based) Updated with each new release of JCR Impact Factors Algorithms and methodology are described at the website: http://www.eigenfactor.org Published in Journal Citation Reports, and at Eigenfactor website • Journal-level metric • Score captures relative importance of a journal on a per article basis, but is not tied to a specific article(s). • Article Influence scores of a journal can vary between eigenfactor.org and JCR even for the same year. • Eigenfactor metrics may take into account some other sources (such as dissertations) besides journals. • Targeting journals in which to publish • Identifying journals relevant to a specific discipline • Measuring a journal’s status
- 2. IUPUI University Library Center for Digital Scholarship: Metrics for Scholarly Products Created by Kellie Kaneshiro & Heather Coates, rev. 5/2015 2 Metric How Calculated Update Frequency Source Keep in mind Use it For h-Index The largest number h such that h publications have at least h citations. Developed to quantify cumulative impact of a scholar’s published works and may also be used as a productivity measure. (citation-based) Timeframe and updates depend on the source. Can be manually calculated using citation databases. Calculated automatically by Web of Science, Scopus, Google Scholar – the number may vary due to Scholar’s broad coverage, smaller databases tend to be more accurate. http://en.wikipedia.org/wiki/H-index • Typically a scholar-level metric; may also be calculated for journals or any other defined set of documents • Bounded by total number of publications • Favors scholars with longer careers • Does not account for author position or number of co-authors • Researchers with common surnames may be better off calculating the h-index manually • Measuring impact of an individual’s publications • Comparing researchers within disciplines i10 Index Number of publications with at least ten citations (citation-based) Scholar Metrics currently cover articles published 2007-2011 and are based on citations from all articles indexed in Google Scholar as of April 1st, 2012. Sources are unclear and subject to changes, and there is little transparency on how Google Scholar calculates this number. http://scholar.google.com/intl/en/scholar/metrics. html Scholar-level metric • Measuring impact for an individual’s publications • Comparing researchers within disciplines Article-level metrics (PLoS) Not a single metric, but a suite of metrics: • Article usage (views & downloads) • Citations • Social networks • Blogs & media coverage • PLoS community input Real-time • Article usage: PLoS, PubMed Central • Citations: PubMed Central, Scopus, CrossRef, Web of Science • Social Networks: CiteULike, Connotea, Twitter, Facebook, Mendeley • Blogs & Media Coverage: Nature Blogs, Research Blogging, Trackbacks • PLoS Community Input: Reader Comments, Reader Notes, Reader Ratings http://article-level-metrics.plos.org/alm-info/ No single metric, so it can be more complex to present in context. • Demonstrating immediate impact of your research across multiple non- traditional communication channels • Benchmarking performance of a particular item against similar items
- 3. IUPUI University Library Center for Digital Scholarship: Metrics for Scholarly Products Created by Kellie Kaneshiro & Heather Coates, rev. 5/2015 3 Metric How Calculated Update Frequency Source Keep in mind Use it For Impact Story Not a single metric, but a suite of metrics: • Article usage (views & downloads) • Citations • Social networks • Blogs & media coverage • PLoS community input Including, but not limited to • PLoS ALM (PLoS, PubMed Central, CrossRef) • Facebook • Slideshare • Github • Wikipedia • CiteULike • Delicious • Mendeley • Dryad • F1000 http://impactstory.org/ • Gathering IDs may not capture everything • Artifacts may be missing some metrics • Number of items on a report is currently limited • Data displayed is not currently CC0 due to licenses with data sources • Demonstrating immediate impact of your research across multiple non- traditional communication channels • Benchmarking performance of a particular item against similar items Journal acceptance rates Proportion of items accepted for publication in the past year N/A Editors (may have to request) • May not be transparent or easily available • Demonstrating potential impact for an unpublished or relatively recent article Visibility N/A Examples: book reviews, links to an item (from a blog, website, etc.), reputation of individuals reviewing/linking to the item, media coverage, use for policy decisions or clinical guidelines, and other impact upon a community or population Varies Sources vary • May be difficult to do a systematic search and capture for this type of information • Difficult to provide context for comparison • Demonstrating broader impact of your research that does not fit into traditional or formal metrics Ownership count (libraries) As indexed in the WorldCat catalog http://www.worldcat.org/ Depends on the contributing libraries, but a record is added every 10 seconds. OCLC, or Online Computer Library Center, Inc., which is “a nonprofit, membership, computer library service and research organization dedicated to the public purposes of furthering access to the world’s information and reducing information costs.” • May not be recognized or valued as an indicator of impact in some fields • Context may be difficult to provide • As library budgets have decreased, libraries are purchasing fewer items. Instead, they are relying on gaining access via interlibrary-loan (ILL). • Demonstrating broad dissemination • Demonstrating value to academic and/or public audiences
- 4. IUPUI University Library Center for Digital Scholarship: Metrics for Scholarly Products Created by Kellie Kaneshiro & Heather Coates, rev. 5/2015 4 Metric How Calculated Update Frequency Source Keep in mind Use it For WorldCat is the largest online public access catalog in the world. It includes 1.8 billion items represented by 270 million records. These records come from 72,000 libraries worldwide in 170 countries and territories. Indexed in major databases A general indication of the quality of a scholarly publication Ex: for biomedical disciplines, indexed for PubMed in MEDLINE by the National Library of Medicine Varies depending on the database Typically, commercial publishers • The criteria for indexing varies by database and may not be transparent • Demonstrating value of publishing in a journal that is new or not yet established Web metrics (views, downloads, shares) Calculated by the repository or website, typically excluding bots Varies, typically real- time Analytic code within the repository system itself • Similar accuracy issues with all web statistics, although many repositories screen out traffic from bots and web crawlers • Demonstrating the reach and impact of the item • Complementing data from other sources to provide overall indication of impact Editorial Board quality Journal website; reputation among your colleagues Varies Colleagues • Can be unreliable or lag behind actual events • Complementing quantifiable metrics Selected Reading 1. Roemer, R. C. & Borchardt, R. (2012). From bibliometrics to altmetrics: A changing scholarly landscape. College & Research Libraries News, 73(10), 596-600. http://crln.acrl.org/content/73/10/596.full 2. Piwowar, H. & Priem, J. (2013). The power of altmetrics on a CV. Bulletin of the American Society for Information Science and Technology, 39(4), 10-13. DOI: 10.1002/bult.2013.1720390405