Thom Kiddle: Responding to the challenge of testing and assessing speaking

Editor's Notes

• #5: First of all let me give you some introductory information: in this plurilingual exam the candidates use English and a second foreign language which is taught in their school. The expected level for English is B2 while it‘s B1 for the second foreign language. The plurilingual exam consists of 2 equally important parts… Therefore the candidates have to demonstrate two different types of speech – from planned to spontaneous. Paramount of the assessment is the successful communication in a given plurilingual context. So the main criterion of the exam is whether the candidate can master this situation in both languages and not whether he or she has attained level B2 in English and B1 in the second foreign language. As I mention on this slide, these two parts are „equal“ so they are also taken equally in consideration for the final assessment. As there are two examiners – one for each language - there are no additional assessors. This decision has been taken by the Austrian ministry of Education and not by the team of CEBS. But, that is why only one assessment grid which is holistic rather than analytic, has been developed by our team for the two examiners. The two examiners have one vote and therefore must agree on a common final assessment in the form of a grade following the exam. So you can see that there were a number of aspects which had to be taken into consideration during the development of the assessment grid. Let me sum them up: First of all, how can we evaluate the performances in two languages which are spoken on a different level? Secondly, how to formulate descriptors which describe as precisely as possible an oral performance in two languages but can, at the same time, be handled by the examiners. And last but not least, how to include not only productive skills but also receptive skills and personal attitudes which enable the candidates to act in a plurilingual context, as a successful interface between other users of language who are monolingual.
• #6: Let me now switch to the different criteria of our assessment grid. The first one is the fulfilment of the task. One aspect that is commonly assessed is whether all content points have been addressed and to what extent. In addition to this, the relevance of what is said, the clarity and the identification with the context/situation are taken into consideration. What is new or different is that the amount of… because in this plurilingual conversation the most important thing is that the interlocutors receive all the desired information. The second criterion,….includes flexibility. spontaneity and initiative – aspects which are very often taken into consideration when you are assessing speaking performances. What makes the difference in our assessment is that the examiners should also take into account to what extent the candidate is able to switch between two languages or even between three languages – because the task itself and most of the input are in German.
• #7: The next two aspects of this criterion are not yet commonly included in assessment grids for speaking. The first one is the ability to mediate between two interlocutors who speak different languages. But mediation does not only mean to pass on information but also enabling interaction between the two interlocutors e.g. explaining terms which are not accessible when they are only translated. The second aspect is awareness of intercultural differences. Brian North and his group of experts are currently working on new descriptors for these two aspects of language performance. So I would say that CEBS made a “first step” when including them into our assessment grid, but after the publication of the work of the European Council/Brian and his group, we will certainly rethink our grid, perhaps adapt or even improve it. Range… is our third criterion and it includes… of structures and vocabulary to allow… Within this criterion compensation strategies, as for example the attempt to transfer words from one language to the other, are taken into consideration even if they are difficult to assess. In addition to that, there is a combination of two different levels (B2 +B1) and so the examiners have to apply descriptors bearing in mind this difference of levels. Nevertheless, the most important thing is that the candidate has mastered the plurilingual situation. At the end of this short presentation I would like to show you what our grid looks like.
• #17: In this 10 minutes we’ll look at four questions, and probably raise more questions than answers.
• #18: Not very much; certainly nothing prescriptive. But the CEFR isn’t meant to be a checklist, or an assessment tool in itself. It can be starting point or a guide, and here the key words are ‘illustrative’ and ‘meaning’. So how to measure ‘both form and meaning’?
• #19: The ISE Speaking & listening rating scale, and also the Writing scale, are specific to the level – A2, B1, B2 and C1. So the performance of an A2 candidate is measured against a scale for A2 candidates. Each scale has four levels, giving an indication of performance that falls above, at or below the CEFR level. The ‘language control’ criterion focusses on what the candidate can do with their linguistic resources, i.e. what language functions they can achieve, rather than which specific tense they can use in an utterance. Also, linguistic competence is part of communicative competence, and is part of the interaction between the test-taker, his or her cognitive processes, the context, the task, the rating scales and how they are interpreted by the rater.
• #20: Collaboratively – all stakeholders were consulted and part of the process; based on evidence – i.e. performances of language learners; academic research; the CEFR; drawn up, trialled, evaluated, adapted; trialled again by examiners, CEFR experts, academics and testing experts. And now they are live, the process of evaluating and testing them continues.