Peer review uncertainty at the institutional level
This document summarizes a study that compared peer review evaluations to bibliometric metrics at the institutional level in Italy. The study analyzed a random sample of over 7,000 publications that underwent peer review as part of the Third Italian Research Evaluation Exercise covering 2011-2014. Peer reviews were conducted at both the article and institutional levels. Metrics included average citation counts, journal impact factors, and citation percentiles. The results showed that correlations between peer review and metrics were higher at the institutional level than the article level. Agreement was also higher for journal indicators than citation indicators. Internal peer review agreement was comparable to agreement with journal indicators, raising the question of whether reviewers base evaluations largely on the journal a paper appears in.
Peer review uncertainty at the institutional level
- 1. Peer review uncertainty at the institutional level V.A. Traag1, M. Malgarini2, T. Cicero2, S. Sarlo2, L. Waltman1 1Centre for Science and Technology Studies (CWTS), Leiden University, the Netherlands 2ANVUR, Rome, Italy
- 2. Comparing peer review with metrics • Post-publication research evaluation • Context: performance based research funding • How well do metrics agree with peer review? • Relevant aggregation level – Institutional level – Article level • Consider peer review uncertainty 1
- 3. Third Italian Research Evaluation Exercise (VQR 2011-2014) • VQR launched in June 2015, covering the period 2011— 2014. • VQR covered • 114 431 publications by • 60 455 researchers working in • 96 Universities, • 12 Public Research Organizations (PRO) , and • 27 other research Institutes (participating on a voluntary basis). • in 16 research areas (GEVs) • of which 11 bibliometric GEVs. • VQR executed by the Italian National Agency for the evaluation of Universities and Research Institutes (ANVUR, ENQA Affiliate from September 2013). 2
- 4. Final Result VQR research outputs evaluation method 3 Peer-review method Informed peer-review method GEV final approval Research Outputs to be evaluated GEV assigns each output to two GEV members GEV members may choose evaluation method on the basis of: • GEV rules (stated into GEV criteria documents) • Output characteristics Excellent Good Fair Acceptable Limited Ineligible
- 5. Sample journal articles • Random sample of 10% of all output of bibliometric GEV. • The sample has been stratified on the basis of GEV. • Selected articles were submitted for peer review using the same process as in the VQR exercise. • The number of articles effectively peer reviewed covers 9.3% of all articles submitted to bibliometric evaluation. • No substantial selection biases (language, number of pages, bibliometric evaluation) emerged from a post stratification analysis. • The final sample includes 7164 publications. 4
- 6. Sample shares at the institutional level 5
- 7. Bibliometric Methodology • Sampling 7164 papers, matched 5183 to WoS. • 4560 from 78 Italian universities, almost 8%. • Two peer reviews of same article – Randomly designated reviewer 1 and reviewer 2 – Score between 3 – 30 (covering originality, rigour and impact) • Institutional scores – Average of peer review scores – Average of normalised WoS citations – Average of normalised WoS journal impact – Average of VQR citation percentile (various data sources) – Average of VQR journal percentile (various data sources) 6
- 8. Metrics and peer review uncertainty • Peer review uncertainty – Compare institutional score of reviewer 2 to reviewer 1. • Metrics – Compare institutional metric score to reviewer 1. 7
- 9. Article Spearman correlations (WoS) 8
- 12. Conclusions • Institutional level agreement higher than article level agreement. • Agreement with peer review is higher for journal indicators than for citation indicators. • Internal peer review agreement comparable to journal indicator agreement. • Do reviewers base evaluation largely on journal? 11