![Are tests doing what they need to?
Contemporary language examination tasks?
• Describe a picture
• Talk about [2 out of 3] pictures for one minute
• Talk about an experience (monologue to a computer)
• Speak about a topic you are given after one minute’s
thinking time (for that topic)
• Listen and repeat sentences
• Listen and answer multiple choice questions about what
you’re listening to
• Listen to a recording and select the correct answers
• Listen and write the correct answers
• Read aloud – words or sentences (to a computer)
• Read and select true, false or not given
• Transform sentences using knowledge of grammar/syntax
• Write about, and compare, pictures, graphs/charts
When did you last do
any of these?
…are they fit for
purpose?
…when would your
learners?](https://image.slidesharecdn.com/alexthorptestingtests-180503085512/85/Alex-Thorp-Testing-tests-Realising-the-potential-of-assessment-practices-45-320.jpg)
Alex Thorp: Testing tests. Realising the potential of assessment practices
- 1. Testing tests. Realising the potential of assessment practices Alex Thorp. Lead Academic, Language (Europe) Trinity College London ©Eaquals Eaquals International Conference | Prague | 26-28 April 2018 #eaquals18prague
- 2. Overview 1. Introduction – Your beliefs 2. Overview of contemporary testing 3. Test analysis framework 4. Case study – testing tests 5. Trinity test task example 6. Review - Assessment literacy 7. Summary
- 3. Testing testing. Our objective Why do you use the tests / exams that you use? What outcome do you want to achieve? How do you establish whether the test / exam is fit for purpose?
- 4. Testing testing. Our objective All stakeholders require a framework to analyse, critically evaluate or design and develop test tasks
- 5. Trinity College London English qualifications for real-world communication 1 – Testing – your beliefs
- 7. Helps language acquisition Helps wider skill development 10 0 0 10 What does assessment achieve?
- 8. Trinity College London English qualifications for real-world communication 2 – Overview of contemporary testing
- 9. Competence Content Contemporary testing Behaviourist Constructivist Our understanding of language proficiency is constantly changing – Developments in: • Assessment theory and application • Language pedagogy and methodology • Neurolinguistics • Applied linguistics • Social, cultural and political contexts
- 10. Competence Content Contemporary testing Behaviourist Constructivist The centrality of the test taker
- 11. Washback Tests influence what and how teachers teach, and what and how students learn. (Hughes 1993. Bailey 1996. Wall 1997)
- 12. Washback Notion of washback can be embedded in test design (Messick 1996 f)
- 13. Placing test-taker at centre of test Test taker Learner resources Contextual parameters Cognitive processes Test consequences Placing the individual test taker at the heart of the language testing agenda. O’Sullivan 2011
- 14. Contemporary framework Test taker Cognitive validity Response Scoring validity Context validity Score / Grade Consequential validity Criterion- related validity (Weir 2005f) An accessible and practical tool to design and/or critically evaluate tests
- 15. Socio-cognitive framework Communication is a social phenomenon set in a context, involving cognitive processes and linguistic resource
- 16. Trinity College London English qualifications for real-world communication 3 – Concerns - summative assessment
- 17. Concerns with Summative assessment Some common pitfalls (there are many more) Misappropriation Prioritisation of score Skills only for test Under representation Construct irrelevant variance Macro-skill complexity Punitive marking Lack of diagnostics Delivery Design
- 18. Trinity College London English qualifications for real-world communication 3 – Test analysis framework
- 19. Test or test task – fit for purpose? The three C’s Candidate characteristics Context validity Cognitive validity Taylor 2017
- 20. Test or test task – fit for purpose? The three C’s Candidate characteristics Context validity Cognitive validity Taylor 2017 Purpose and Format Linguistic context and socio-cultural context to task (Setting and Demands Cognitive processing and information sources called upon – relevance to real- world What individual brings to test Physical / Experiential / Psychological
- 21. Test or test task – fit for purpose? Test analysis tool Analyse reliability (scoring) Analyse anticipated response Analyse test features Define purpose
- 22. Assessment – Task analysis Test-taker characteristics, Context and Cognitive validity
- 23. Fit for purpose? Task analysis Candidate characteristics - Example questions To use background knowledge and engage schemata through personalisation To engage translanguaging practices where beneficial, bridging L1 and L2 To contextualise learning within relevant local setting, familiar to test-taker To be sensitive to affective influences of learner, focus on learning not outcome Physical – Psychological - Experiential
- 24. Fit for purpose? Task analysis Context validity - Example Task setting – task demands – language demands Task demands - define Domain: Personal / educational Discourse mode – Descriptive, instructive, expository, argumentative Content knowledge – general or specific topic Information type – Concrete of abstract, familiarity
- 25. Fit for purpose? Task analysis Context validity - Example Task setting – task demands – language demands Language demands Range; Sufficient to express self around familiar topics Accuracy; Reasonably accurate around frequently used routine language and patterns Fluency; Can keep going comprehensibly, some hesitation Interaction; Can initiate, maintain and close simple face-to- face conversation Coherence; Can link series of shorter, discrete sentences
- 26. Fit for purpose? Task analysis Context validity - Example Task setting – task demands – language demands Language demands – communication skills (Mapped to CEFR B1) Can reasonably fluently sustain a straightforward description of subjects within field of interest Can generally follow the main points of extended discussion Can exploit a wide range of simple language to deal with most situations likely to arise whilst travelling Can enter unprepared into conversations on familiar topics and follow clearly articulated speech Can maintain a conversation or discussion but may sometimes be difficult to follow Can express and respond to feelings such as surprise, happiness, sadness and interest Can give or seek personal views and opinions in discussing topics of interest Can make opinions and reactions understood as regards solutions to problems or practical questions Can express belief, opinion, agreement and disagreement politely
- 27. Fit for purpose? Task analysis Cognitive validity - Example Physical – Psychological - Experiential Cognitive processes: to practice and develop relevant cognitive processes (Field 2011) Conceptualisation (speaker goal, world knowledge, narrative recall) Grammar encoding (syntax, collocational competence, topic recall) Morpho-phonological encoding (Lexical and phonological knowledge) Phonetic encoding (articulation and production) Self-monitoring (assess target utterance, fulfil speaker goal)
- 28. Trinity College London English qualifications for real-world communication 4 – Case study – Testing tests
- 29. Fit for purpose?
- 30. Test or test task – fit for purpose? Test analysis tool Analyse reliability (scoring) Analyse anticipated response Analyse test features Define purpose
- 31. Bachman & Palmer (2010): communicative competence model Communicative Competence Linguistic competence Socio-linguistic competence Discourse competence Strategic competence Communicative competence
- 32. Test or test task – fit for purpose? A proficiency task – communicative exam Example task – at CEFR B1 A stimulus task for assessment of communicative competence, subjectively assessed with a standardised rating criteria Compare and contrast the two images and say why people spend their free-time doing these activities
- 33. Fit for purpose? Task analysis Candidate characteristics - Example questions To use background knowledge and engage schemata through personalisation To engage translanguaging practices where beneficial, bridging L1 and L2 To contextualise learning within relevant local setting, familiar to test-taker To be sensitive to affective influences of learner, focus on learning not outcome Physical – Psychological - Experiential
- 34. Fit for purpose? Task analysis Context validity - Example Task setting – task demands – language demands Language demands Range; Sufficient to express self around familiar topics Accuracy; Reasonably accurate around frequently used routine language and patterns Fluency; Can keep going comprehensibly, some hesitation Interaction; Can initiate, maintain and close simple face-to- face conversation Coherence; Can link series of shorter, discrete sentences
- 35. Fit for purpose? Task analysis Context validity - Example Task setting – task demands – language demands Language demands – communication skills (Mapped to CEFR B1) Can reasonably fluently sustain a straightforward description of subjects within field of interest Can generally follow the main points of extended discussion Can exploit a wide range of simple language to deal with most situations likely to arise whilst travelling Can enter unprepared into conversations on familiar topics and follow clearly articulated speech Can maintain a conversation or discussion but may sometimes be difficult to follow Can express and respond to feelings such as surprise, happiness, sadness and interest Can give or seek personal views and opinions in discussing topics of interest Can make opinions and reactions understood as regards solutions to problems or practical questions Can express belief, opinion, agreement and disagreement politely
- 36. Fit for purpose? Task analysis Cognitive validity - Example Physical – Psychological - Experiential Cognitive processes: to practice and develop relevant cognitive processes (Field 2011) Conceptualisation (speaker goal, world knowledge, narrative recall) Grammar encoding (syntax, collocational competence, topic recall) Morpho-phonological encoding (Lexical and phonological knowledge) Phonetic encoding (articulation and production) Self-monitoring (assess target utterance, fulfil speaker goal)
- 37. Test or test task – fit for purpose? A proficiency task – communicative exam Example task – at CEFR B1 A stimulus task for assessment of communicative competence, subjectively assessed with a standardised rating criteria Compare and contrast the two images to your partner Not a ‘bad’ task de facto A question of being appropriate The danger is misappropriation
- 38. Trinity College London English qualifications for real-world communication 5 – Trinity Integrated Skills in English
- 39. Test or test task – fit for purpose? Example task at B1 - Communicative Example task – at CEFR B1 A stimulus task for assessment of communicative competence, subjectively assessed with a standardised rating criteria
- 40. Test or test task – fit for purpose? Test analysis tool Analyse reliability (scoring) Analyse anticipated response Analyse test features Define purpose
- 41. Fit for purpose? Task analysis Candidate characteristics - Example questions To use background knowledge and engage schemata through personalisation To engage translanguaging practices where beneficial, bridging L1 and L2 To contextualise learning within relevant local setting, familiar to test-taker To be sensitive to affective influences of learner, focus on learning not outcome Physical – Psychological - Experiential
- 42. Fit for purpose? Task analysis Context validity - Example Task setting – task demands – language demands Language demands Range; Sufficient to express self around familiar topics Accuracy; Reasonably accurate around frequently used routine language and patterns Fluency; Can keep going comprehensibly, some hesitation Interaction; Can initiate, maintain and close simple face-to- face conversation Coherence; Can link series of shorter, discrete sentences
- 43. Fit for purpose? Task analysis Context validity - Example Task setting – task demands – language demands Language demands – communication skills (Mapped to CEFR B1) Can reasonably fluently sustain a straightforward description of subjects within field of interest Can generally follow the main points of extended discussion Can exploit a wide range of simple language to deal with most situations likely to arise whilst travelling Can enter unprepared into conversations on familiar topics and follow clearly articulated speech Can maintain a conversation or discussion but may sometimes be difficult to follow Can express and respond to feelings such as surprise, happiness, sadness and interest Can give or seek personal views and opinions in discussing topics of interest Can make opinions and reactions understood as regards solutions to problems or practical questions Can express belief, opinion, agreement and disagreement politely
- 44. Fit for purpose? Task analysis Cognitive validity - Example Physical – Psychological - Experiential Cognitive processes: to practice and develop relevant cognitive processes (Field 2011) Conceptualisation (speaker goal, world knowledge, narrative recall) Grammar encoding (syntax, collocational competence, topic recall) Morpho-phonological encoding (Lexical and phonological knowledge) Phonetic encoding (articulation and production) Self-monitoring (assess target utterance, fulfil speaker goal)
- 45. Are tests doing what they need to? Contemporary language examination tasks? • Describe a picture • Talk about [2 out of 3] pictures for one minute • Talk about an experience (monologue to a computer) • Speak about a topic you are given after one minute’s thinking time (for that topic) • Listen and repeat sentences • Listen and answer multiple choice questions about what you’re listening to • Listen to a recording and select the correct answers • Listen and write the correct answers • Read aloud – words or sentences (to a computer) • Read and select true, false or not given • Transform sentences using knowledge of grammar/syntax • Write about, and compare, pictures, graphs/charts When did you last do any of these? …are they fit for purpose? …when would your learners?
- 46. Trinity College London English qualifications for real-world communication 4 – Summary
- 47. Fit for purpose? Assessment literacy Summary • No such thing as a ‘valid’ task • Task or test only valid for purpose it was designed for • The test-taker is at the heart of the task • Stakeholders are responsible for practices and effect • Stakeholders should seek positive washback • A contemporary framework can be used to critically assess tasks or tests • Assessment literacy skills can unlock learning potential of assessment practices
- 48. Summary 1. Introduction – Your beliefs 2. Overview of contemporary testing 3. Test analysis framework 4. Case study – testing tests 5. Trinity test task example 6. Review - Assessment literacy 7. Summary
- 49. Trinity College London Q&A Any Questions? For a copy of slides / a paper on Testing tests please email: Alex.thorp@trinitycollege.com
- 50. Selected references Bachman, L F. (1990), Fundamental Considerations in Language Testing. OUP Brown, H. D. (2004). Language assessment: Principles and classroom practices. New York: Pearson/Longman. Canale, M. and Swain, M. (1980) Theoretical bases of communicative approaches to second language teaching and testing. Applied linguistics, 1, 1- 47. O’Sullivan, B., & Weir, C. J. (2011). Test development and validation. Language testing: theories and practices. London: Palgrave Macmillan. Papageorgiou, S. (2007) Relating the Trinity College London GESE and ISE examinations to the Common European Framework of Reference Underhill, N. (1987). Testing spoken language: A handbook of oral testing techniques. Cambridge University Press. Weir, C. J. (2005). Language testing and validation. Hampshire: Palgrave McMillan.