Replacing peer review by metrics in the UK REF?
2 likes290 views
The document discusses using citation metrics as an alternative to peer review for assessing research quality in the UK's Research Excellence Framework (REF). It finds high correlations (0.69-0.86) between citation metrics and peer review ratings at the institutional level across various fields like physics and chemistry. It also models peer review uncertainty and finds citation metrics correlate almost as well as the peer review model for physics. The authors conclude metrics could potentially replace peer review for the REF, pending further analysis.
Replacing peer review by metrics in the UK REF?
- 1. Replacing peer review by metrics in the UK REF? CWTS, Leiden University 8 September 2017 V.A. Traag & L. Waltman
- 2. UK REF • Research Excellence Framework (REF) is UK’s system “to assess the quality of research” of institutions. • Each institution assessed separately per Unit of Assessment (UoA) e.g. Physics, Chemistry, etc…. • Three profiles are assessed: 1 Research output; 2 Impact beyond academia; and 3 Research environment. • Each output is awarded 1–4 stars. 4∗ is “world-leading”, 1∗ is “recognised nationally”. • Publications are awarded 1–4 stars based on peer review. • Results are divulged at institutional/UoA level (as % of 1–4∗). 2/18
- 3. Metrics • Metric Tide (2015) correlate peer review results with various citation metrics on article level. Find correlations roughly 0.3 – 0.6. • Myrglod et al. (2015) use departmental h-index, correlations range from 0.39 (Physics) – 0.71 (Chemistry). • Elsevier (2015) use proportion highly cited publications. Find correlations roughly 0.20 (Physics) – 0.75 (Chemistry). 3/18
- 4. Problems • Correlation analysis of Metric Tide at wrong level. Should be institutional/UoA level, not article level. Possible to have low correlations at lower level but high correlations at higher level. • Difference in set of publications. Elsevier & Myrglod based on coverage in Scopus subject categories of UoA. • Central problem: when is a correlation high enough? Outliers. • Peer review also has uncertainties. 4/18
- 5. Analysis • Re-do citation analysis of all submitted publications. • Calculate correlations at institutional level. • Compare to uncertainty in peer-review. Unfortunately, peer review results per article are unavailable. Create model of peer review uncertainty. Central idea: what if we repeat the REF? 5/18
- 6. Citation analysis • Match all publications to WoS using DOI. • Use citation window until 2014 (publications from 2008–2014). • Normalise based on CWTS publication classification. • Calculate PP(top 10%) indicator per institution/UoA. • Correlations between PP(top 10%) and PP(4∗) per UoA. 6/18
- 7. Correlation results 10 20 30 40 50 60 70 PP(Top 10%) 0 5 10 15 20 25 30 35PP(4*) Correlation = 0.86 Unit of Assessment: Physics 7/18
- 8. Correlation results 0 10 20 30 40 50 60 PP(Top 10%) 10 0 10 20 30 40 50PP(4*) Correlation = 0.83 Unit of Assessment: Chemistry 8/18
- 9. Correlation results 10 20 30 40 50 60 70 PP(Top 10%) 10 0 10 20 30 40 50 60PP(4*) Correlation = 0.69 Unit of Assessment: Biological Sciences 9/18
- 10. Review Model 1 Each paper has some value vi. 2 Review yields some perceived value pi = vi . 3 Award 4∗ if perceived value exceeds some threshold pi > p4∗ . Resampling 1 Sample a value vi given a paper’s 4∗ status. 2 Sample perceived value pi = vi . 3 Award 4∗ if perceived value exceeds some threshold pi > p4∗ . 10/18
- 11. Probability 4∗ 0.0 0.5 1.0 1.5 2.0 Value 0.0 0.2 0.4 0.6 0.8 1.0(*=) = = . = . 11/18
- 12. Probability of value (σ2 ) 0.0 0.5 1.0 1.5 2.0 2.5 3.0 0.00 0.05 0.10 0.15 0.20 0.25 0.30(=*) = = . = . 12/18
- 13. UK REF Correlations EconomicsandEconometrics ClinicalMedicine Physics Chemistry PublicHealth,HealthServicesandPrimaryCare EarthSystemsandEnvironmentalSciences Psychology,PsychiatryandNeuroscience ElectricalandElectronicEngineering,MetallurgyandMaterials BiologicalSciences Agriculture,VeterinaryandFoodScience Geography,EnvironmentalStudiesandArchaeology BusinessandManagementStudies PoliticsandInternationalStudies GeneralEngineering AlliedHealthProfessions,Dentistry,NursingandPharmacy ComputerScienceandInformatics Communication,CulturalandMediaStudies,LibraryandInformationManagement Philosophy Education ArtandDesign:History,PracticeandTheory Law History MathematicalSciences Aeronautical,Mechanical,ChemicalandManufacturingEngineering SocialWorkandSocialPolicy SportandExerciseSciences,LeisureandTourism TheologyandReligiousStudies EnglishLanguageandLiterature Music,Drama,DanceandPerformingArts Architecture,BuiltEnvironmentandPlanning CivilandConstructionEngineering Sociology ModernLanguagesandLinguistics Classics AreaStudies AnthropologyandDevelopmentStudies 0.0 0.2 0.4 0.6 0.8 1.0Correlation PP(top 10%) Bootstrap (0.1) Bootstrap (0.5) Bootstrap (1) trics icine ysics istry Care nces ence rials 0.0 0.2 0.4 0.6 0.8 1.0 Correlationtrics icine ysics istry Care nces ence rials nces ence 0.0 0.2 0.4 0.6 0.8 1.0 PP(to 13/18
- 14. Conclusions Conclusions • Correlations at article level = correlations at institutional level. • Correlations on metrics should consider peer review uncertainty. • For UK REF, physics metrics correlate almost equally well as (model of) peer review. Future • Longer citation window? • Compare to Scopus correlations. • Corroborate model of peer review uncertainty. 14/18
- 16. Model technicalities Distributions • Value distributed as vi ∼ LogN(µv , 1). • Error distributed as ∼ LogN(−σ2 2 , σ2), so that = 1. • Perceived value distributed as pi ∼ LogN(µv − σ2 2 , 1 + σ2). Estimation • Assume overall value µv = 1. • Calculate p4∗ such that Pr(pi > p4∗ ) = PP(4∗) overall. • Assume values for institution/uoa is LogN(µv , 1). • Calculate µv such that Pr(pi > p4∗ ) = PP(4∗) per institution/uoa. 16/18
- 17. Probability of value (µv) 0 1 2 3 4 5 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45(=*) = . = = 17/18
- 18. Accuracy 0.0 0.2 0.4 0.6 0.8 1.0 PP(4*) 0.0 0.2 0.4 0.6 0.8 1.0Accuracy Accuracy (4*) Accuracy (< 4*) Accuracy (overall) 18/18