Measure for Measure: The role of metrics in assessing research performance - Society for Scholarly Publishing. June 2013
8 likes7,140 views
The document discusses the role of metrics in assessing research performance. It notes that the choice of metric depends on the level and type of impact being assessed, such as article, journal, researcher or institution. While impact factor is the most widely known metric, the document recommends using a variety of metrics to provide a richer view. It promotes transparent, standard metrics like Scopus Cited-by counts, SNIP and SJR which can be easily embedded. The document also notes that awareness of metrics affects their acceptance and recommends raising awareness of newer alternative metrics.
Measure for Measure: The role of metrics in assessing research performance - Society for Scholarly Publishing. June 2013
- 1. Measure for Measure The role of metrics in assessing research performance Society for Scholarly Publishing - June 2013 MichaelHabib, MSLS Product Manager, Scopus habib@elsevier.com Twitter: @habib http://orcid.org/0000-0002-8860-7565
- 2. 1. What level am I assessing? – Article, Journal, Researcher, Institution, etc. Which metric should I use?
- 3. 1. What level am I assessing? – Article, Journal, Researcher, Institution, etc. 2. What type of impact am I assessing? – Research, Clinical, Social, Educational, etc. Which metric should I use?
- 4. 1. What level am I assessing? – Article, Journal, Researcher, Institution, etc. 2. What type of impact am I assessing? – Research, Clinical, Social, Educational, etc. 3. What methods are available based on above? – Metrics: Citation, Usage, Media, h-index, SNIP, SJR, etc. – Qualitative: Peer-Review, etc. Which metric should I use?
- 6. Article-Level-Metrics Altmetrics Image from: http://www.plosone.org/article/metrics/info%3Adoi% 2F10.1371%2Fjournal.pone.0006022 APIs are available to easily + freely embed Scopus Cited-by countson your Article pages
- 8. Background & approach Who & when: 54,442 individuals were randomly selected from Scopus. They were approached to complete the study in October2012.To ensure an unbiasedresponseElsevier’sname was only revealed atthe end of the survey. Responses:The online survey took around 15-20 minutes to complete. 3,090 respondentscompletedit, representing a response rate of 5.7%. Data has not beenweighted. There was a representative response by country and discipline. Statisticaltesting: Error margin ± 1.5%, at 90% confidencelevels.When comparing the score formain group and sub-groups we have used a Z test of proportion to identify differencesbetweenthe overallaverage and the sub-group (90% confidence levels),when there are 30 or more responses. 8 Adrian Mulligan, Gemma Deakin and Rebekah Dutton Elsevier Research & Academic Relations
- 9. 9 Most widely known by researchers Impact Factor (n=2,520)* 82% H-Index (n=1,335) 43% Journal Usage Factor (n=309) 10% Altmetrics (n=41)* 1% Impact Factor is published by Thomson Reuters, Altmetrics were least well known
- 10. Awarenes s 10 Q2 Which of these do you think are most useful at measuring research quality? (Select up to 3) 64% 29% 29% 28% 58% 37% 34% 42% 0% 20% 40% 60% 80% 100% Impact factor (n=2,530) SNIP (n=51) SJR (n=126) Eigenfactor (n=285) h-index (n=1,335) Journal Usage Factor (n=309) F1000 (n=155) Alt-metrics (n=41) * Only people who said they were aware of a particular metric in Q1 were given the opportunity to select that metric in Q2, *See appendix for background and approach.Research by Elsevier Research & Academic Relations. ImpactFactor is published by Thomson Reuters, TOTAL (n=3,090) 82% 2% 4% 9% 43% 10% 5% 1% Researcher perception of most useful % useful
- 11. Generally, metrics with the highest awareness are also considered to be the most useful 11 Impact factor SNIP SJR Eigenfactor h-index Journal Usage Factor F1000 Alt-metrics R² = 0.6972 0% 10% 20% 30% 40% 50% 60% 70% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Percentageofawarerespondentsthatchosethe metricasoneofthemostuseful Percentage of respondents that are aware of the metric The trendline showsthe linear trendfor the relationship betweenawarenessandusage of metrics Metricsabove the line have lowerlevelsof awareness,but are more likelytobe ratedas useful thanthe typical awareness-usage relationship Metricsbelowthe line have higherlevelsof awareness, butare lesslikelytobe rated as useful thanthe typical awareness-usage relationship *See appendix for background and approach.Research by Elsevier Research & Academic Relations. ImpactFactor is published by Thomson Reuters,
- 12. 13 Assessing the usefulness of potential quality metrics: by age Significantdifferencebetween subsetandtotal (subset higher) Significantdifferencebetween subsetandtotal (subset lower) Under 36 (n=540) 36-45 (n=920) 46-55 (n=819) 56-65 (n=507) Over 65 (n=242) TOTAL (n=3,090) Article views/downloads (for articles) 43% Citations from materials thatarein repositories 43% Share in social network mentions (for articles) 16% Number of readers (for articles) 40% Number of followers (for researchers) 31% Votes or ratings (for articles) 24% A metric that measures the contribution an individual makes to peer review (for researchers) 28% A score basedon reviewer assessment (for articles) 28% Q3 Thinking about possible new measures of research productivity, how useful do you think the below would be in assessing the quality of a researcher or a researcharticle?(By age) % Think it would be extremely/veryuseful 43% 49% 21% 42% 38% 35% 34% 33% 44% 45% 18% 41% 33% 24% 29% 29% 45% 41% 15% 39% 28% 22% 27% 27% 44% 41% 12% 41% 30% 22% 26% 27% 36% 37% 13% 35% 30% 19% 24% 27%
- 13. 14 Assessing the usefulness of potential quality metrics: by region (1 of 2) Significantdifferencebetween subsetandtotal (subset higher) Significantdifferencebetween subsetandtotal (subset lower) Africa (n=72) APAC (n=803) Eastern Europe (n=183) Latin America (n=182) TOTAL (n=3,090) Articleviews/downloads (for articles) 43% Citations frommaterials thatare in repositories 43% Share in social network mentions (for articles) 16% Number of readers (for articles) 40% Number of followers (for researchers) 31% Votes or ratings (for articles) 24% A metric that measures the contribution an individual makes to peer review (for researchers) 28% A scorebased on reviewer assessment(for articles) 28% Q3 Thinking about possible new measures of research productivity, how useful do you think the below would be in assessing the quality of a researcher or a research article? (By region, slide 1 of 2) % Think it would be extremely/very useful 56% 51% 26% 49% 36% 33% 40% 44% 50% 55% 27% 46% 46% 29% 35% 36% 50% 49% 19% 45% 41% 30% 28% 26% 50% 49% 21% 45% 34% 24% 32% 35%
- 14. 15 Assessing the usefulness of potential quality metrics: by region (2 of 2) Significantdifferencebetween subsetandtotal (subset higher) Significantdifferencebetween subsetandtotal (subset lower) Middle East (n=47) North America (n=770) Western Europe (n=1,033) TOTAL (n=3,090) Articleviews/downloads (for articles) 43% Citations frommaterials that are in repositories 43% Share in social network mentions (for articles) 16% Number of readers (for articles) 40% Number of followers (for researchers) 31% Votes or ratings (for articles) 24% A metric that measures the contribution an individual makes to peer review (for researchers) 28% A scorebased on reviewer assessment(for articles) 28% Q3 Thinking about possible new measures of research productivity, how useful do you think the below would be in assessing the quality of a researcher or a research article? (By region, slide 2 of 2) % Think it would be extremely/very useful 40% 40% 19% 43% 32% 28% 32% 34% 41% 42% 10% 36% 23% 19% 26% 26% 36% 32% 11% 36% 23% 22% 23% 22%
- 15. “For publishers Greatly reduce emphasis on the journal impact factor as a promotional tool, ideally by ceasing to promote the impact factor or by presenting the metric in the context of a variety of journal- based metrics … that provide a richer view of journal performance.” – from The San Francisco Declaration on Research Assessment(DORA)(http://am.ascb.org/dora/ )
- 16. Transparency: Calculated by independent third-parties; freely and publicly accessible www.journalmetrics.com Subject Field Normalization: allows for comparison independent of the journals’ subject classification. Reflects most current journal scopes, thereby taking ongoing changes into account 3-year citation window: demonstrably the fairest compromise Manipulation-resistant: Article type consistency. Only citations to and from articles, reviews, and conference papers are considered Breadth of coverage: Scopus has over 20,500 sources: 19,500 journals as well as trade publications, proceedings and book series. Metrics based on Scopus.com: underlying database available for transparency; Titles indexed based on transparent criteria by independent advisory board CONS: More complex methodology Do not take amount of review content into account Low awareness Advantages of SNIP & SJR
- 17. Modified SNIP • Refined metric calculation,bettercorrects for field differences • Outlier scores are closer to average • Readilyunderstandablescoringscale with an average of 1 for easy comparison Modified SJR • More prestigious nature of citationsthat come from within the same, or a closely related field • Overcome the tendency for prestige scores the quantityof journalsincreases • Readily understandablescoring scale with an average of 1 for easy comparison http://www.journalmetrics.com/
- 18. + + + A journal’s raw impact per paper Citation potential in its subject field Peer reviewed papers only A field’s frequency and immediacy of citation Database coverage Journal’s scope and focus Measured relative to database median SNIP: Source-normalized impact per paper
- 19. SNIP: Molecular Biology VS Mathematics Journal RIP Cit. Pot. SNIP (RIP/Cit. Pot.) Inventionesmathematicae 1.5 0.4 3.8 MolecularCell 13.0 3.2 4.0
- 20. APIs are available to easily + freely embed these metrics on your journal homepages
- 22. Snowball Metrics … • Support universities’ strategic decision making processes • Aim to encompass the entire scope of key research and enterprise activities • What is the driver? – universities’ metrics tend to suit their data and priorities. With Snowball, they agree to a single method so that they can benchmark themselves against their peers • What is special about them? – owned by distinguished universities, including Oxford and Cambridge, and not imposed by e.g. funders. Universitiestaking control of their own destiny! – Triedand tested methodologiesthat are available free-of-charge to the higher education sector – Academia– industry collaboration 23
- 23. Vision for Snowball Metrics Snowball Metrics drive quality and efficiency across higher education’s research and enterprise activities, regardless of system and supplier, since they are the preferred standards used by research-intensive universities to view their own performance within a global context 24 Snowball Metrics Project Partners
- 24. Snowball Metrics Recipe Book 25 Agreed and tested methodologies for new Snowball Metrics, and versions of existing Snowball Metrics, are and will continue to be shared free-of-charge. None of the project partners will at any stage apply any charges for the methodologies. Any organisation can use these methodologies for their own purposes, public service or commercial. (Extracts from Statement of intent,October2012) www.snowballmetrics.com/metrics
- 25. 1. Choose methods + metrics appropriate to level and impact type being assessed (DORA) 2. Don’t confuse level with type (alms ≠ altmetrics) Free + easy to embed Scopus Cited-by counts on article pages http://www.developers.elsevier.com/ 3. Awareness of metrics correlates to acceptance, raising awareness matters 4. APAC + younger researchers open to new metrics 5. Don’t use just one metric, promote a variety of metrics Free + easy to embed SNIP/SJR on journal homepages http://www.journalmetrics.com/ 6. Choose transparent and standard methods + metrics Learn more about Snowball Metrics http://www.snowballmetrics.com/ In summary
- 26. Michael Habib, MSLS Product Manager, Scopus habib@elsevier.com Twitter: @habib http://orcid.org/0000-0002-8860-7565 http://www.mendeley.com/profiles/michael-habib/ Thank you!