What responsible research assessment DOESN'T mean

Today I was privileged to give a keynote at the Federation of Finnish Learned Societies' annual conference where the theme was Responsible Research Assessment. In light of some concerns bubbling up about assessment reform in the region, I was invited to talk about what responsibility in assessment actually means. Which I did. But I also took the opportunity to outline what responsibility in assessment does NOT mean. Here's what I said:

Responsible does not mean perfect

The first thing we need to acknowledge is that responsible doesn’t mean perfect. As I’ve often argued, there is no such thing as a perfect research assessment. There is no perfect proxy for research quality, or impact, and no perfect mechanism by which we can get to the ‘truth’ as to which researcher deserves the job, and which journal article is ‘better’ than another, or which research group should win the grant.

There is no truth.

Indeed, the introduction of lotteries in grant allocation and other forms of assessment speaks to exactly this point: sometimes there is no clear winner, and we need to be honest about that.

We do assessment reform a disservice if we over-promise what it can deliver. Having said that, we do have a tendency to hold up new assessment approaches to a higher standard than we do our existing approaches. We’re very eager to point out the dangers of newer, more qualitative forms of assessment, without acknowledging the devastation wrought by a massively over-quantified regime.

We can definitely do better than what we've got. And if we use responsible research assessment frameworks like SCOPE, we can make vast improvements to the assessment of research and researchers to everyone’s benefit. But we should be asking not if any resulting assessments are perfect, but if they are  better.

Responsible does not mean quick

The second thing that responsibility in assessment does not mean is that new assessments will be any quicker. When DORA was launched, John Tregoning wrote a piece for Nature in which he cried, ‘Stop saying that publication metrics don’t matter and tell us what does!’. And I think there was an expectation in those early days that we’d just be switching out one quick & dirty indicator for another quick and dirty indicator that did a better job.

But of course we know better now and have recognised that to do this well, and to do it qualitatively, and to recognise a broader range of contributions, is going to take more time, not less. And that is why we need the first principle of SCOPE to evaluate only where necessary. We need to evaluate less frequently but with greater nuance & specificity when we do.

Responsible does not mean that everyone will be happy

The third thing responsibility in assessment does not mean is that everyone will be happy. At a recent conference on research assessment at the University of Lisbon an audience member complained to me that he was a ‘victim of CoARA’. The reason he identified as such was because his project was not funded under a recent funding call whose criteria had changed in line with CoARA.

If we are changing the assessment criteria, different skill-sets and groups will be rewarded. This is a natural consequence of the process.

We can’t assess the success of new measures in accordance with whether they are able to replicate the old measures because the new measures are meant to be doing different things. That’s not to say we shouldn’t be supporting existing and established members of the research community to reframe their contributions in ways that will be well received by funders and institutions, or that we shouldn’t be sympathetic to the plight of early career researchers such as those in the Netherlands who are concerned that new assessment mechanisms may affect their global mobility.

But if we’re changing the way that prizes are handed out we should expect different entities to win those prizes and that, in the short term, there might be some unhappiness about that from those who could formerly guarantee themselves a win under the previous regime.

Responsible does not mean that everyone will agree

And finally, seeking responsibility in assessment does not mean that everyone will agree. We can, and should, seek to co-design & co-interpret our assessments but as we all know: if you put five academics in a room, you will get 10 different points of view.

This is normal.

For this reason, in a recent think-piece for the LSE Impact Blog a group of us who work in research assessment proposed the concept of needing to be 'as aligned as possible but as diverse as necessary' when it came to assessment reform, borrowing from the ‘as open as possible, as closed as necessary’ adage from the open research movement.

We do need a core of agreement around what we value, and how it might be fairly assessed, in order to make life easier for time-strapped, globally-mobile, researchers. However, we also need to respect the autonomy of different regions and disciplines to deliver assessments that align with their own mission and values.

The need to work together to move forward

One thing that we must carefully protect is the research community's ownership of assessment reform. Disagreements, concerns, and expectations, must all be openly discussed, negotiated, and owned, as we seek acceptable ways forward. Be in no doubt that if we don't do it, unappointed, unaccountable, commercial third parties will. In fact, many are already responsible-washing their bibliometric-based offerings as meeting the broader requirements of assessment reform. 

This is a research community mission. And in our efforts to reform research assessment, we must be clear about what such reforms can and can't offer, and what they may and may not look like, as we journey together towards a better and fairer future.