Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s
Download as PPTX, PDF5 likes3,884 views
This document discusses blueprinting and choosing appropriate assessment tools to evaluate student performance. It defines blueprinting as specifying the elements of performance to be represented on an assessment based on their importance. The presentation reviews designing a blueprint by defining the assessment purpose, tabulating curricular content, weighting domains, and deciding the number of assessment items. It also discusses assessing competencies like the CanMEDS 7 roles and ACGME competencies using tools such as in-training evaluations, MCQs, OSCEs, and simulations. Challenges in assessing competencies and potential solutions like linking competencies to observable behaviors are reviewed. The importance of defining assessment purpose and matching it with reliable and valid tools from a created toolbox is emphasized.
Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s
- 1. Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s Claire Touchie, MD, FRCPC Associate Professor, University of Ottawa Chief Medical Education Advisor Medical Council of Canada Property of the Medical Council of Canada
- 3. Objectives By the end of the workshop, participants will be able to: • Define blueprinting in the context of specific assessment needs • Design an appropriate blueprint for the purpose of a specific assessment • Describe available tools for student performance assessments • Construct an assessment tool box to achieve assessment goals 3
- 4. Agenda • Assessment and blueprinting • Exercise #1: Designing a blueprint • Group Discussion • Leg stretch • Assessment tool boxes • Exercise #2: Assembling a tool box • Group Discussion • Wrap-up and Evaluations 4
- 6. Professional Competence Developmental Impermanent Context dependent 6
- 7. Dimensions of Professional Competence Cognitive Habits Technical Of Mind Professional Competence Affective Integrative Moral Contextual Relationship Epstein and Hundert, JAMA 2002; 287:226-235 7
- 8. The CanMEDS Seven C’s Royal College of Physicians and Surgeons of Canada 8
- 9. Charting the Seven C’s Continuum Curriculum Comprehensive Seven Clinical C’s Competency- Based Collaborative Case-based Epstein and Hundert, JAMA 2002; 287:226-235 9
- 10. Purposes of Assessment • Certify the competence of future practitioners • Minimal level of competence • Discriminate among candidates for advanced training • Rank competitively • Provide motivation and direction for learning • Provide formative feedback • Judge the adequacy of a training program • Evaluate teacher or courses Epstein and Hundert, JAMA 2002; Crossley et al, Medical Education, 2002 10
- 11. Purposes of Assessment Learning • To discover the worth • To improve the quality Feedback Assessment 11
- 12. Assessment Does Professional Behavior Authenticity Shows How Knows How Cognition Knows Modification of Miller from Crossley et al. Medical Education, 2002 12
- 13. “If you don’t know where you’re going, you probably ain’t going to get there” Microsoft clip-art, 2012 13
- 14. Blueprinting Defining the “what” should be represented on your assessment • For licensure or certification • Practice analysis • For a course or rotation • Objectives 14
- 15. Blueprint “Specifies all the elements of performance relevant to the assessment so that appropriate samples of activity and corresponding methods can be selected according to their relative importance to the overall assessment process.” Newble & Dawson, 1994 “When the items of a test are judged to adequately represent well-defined domains of content…generalizable samples…” Cronbach, 1971 15
- 16. Domains to be tested Items tested 16
- 17. Blueprint Valid assessment: interpretation of the results reflect what was meant to be tested • Appropriate blueprint • Defines the content of the assessment • Provides validity evidence 17
- 18. How do I blueprint for my assessment? 1. Define the purpose of your assessment • Summative or Formative? • To decide on competency or to rank? • To provide feedback to the student or to the teacher? 2. Tabulate curricular content • Topics vs clinical presentation vs objectives 18
- 19. Example of curricular content • Community acquired pneumonia • Hospital acquired pneumonia • Otitis media • Urinary tract infections • Meningitis • Endocarditis • Osteomyelitis/septic arthritis • Septicemia • Cellulitis/skin infections • HIV 19
- 20. How do I blueprint for my assessment? 3. Provide relative weighting of the content Importance X frequency • Importance/impact 1. Non-urgent, little prevention potential 2. Serious, but not immediately life threatening 3. Life threatening emergency and/or high potential for prevention impact • Frequency 1. Rarely seen 2. Relatively common 3. Very common 20
- 21. How do I blueprint for my assessment? 4. Sample opinion on weighting • Involve course chairs, teachers, evaluation coordinators, previous learners • Consensus method • Readjust weighting if necessary 21
- 22. Example of weighting Content presentation Importance Frequency IXF Weight CAP 3 3 9 0.22 HAP 3 1 3 0.073 Otitis media 1 3 3 0.073 UTI 2 3 6 0.15 Meningitis 3 1 3 0.073 Endocartitis 3 1 3 0.073 Osteo/Septic arthritis 2 1 2 0.049 Septicemia 3 2 6 0.15 Cellulitis/Skin infn 2 2 4 0.098 HIV 2 1 2 0.049 Total 41 22
- 23. How do I blueprint for my assessment? 5. Decide total number of items on your assessment • Based on your weighting, you can decide how many items per content area 23
- 24. How do I blueprint for my assessment? 6. Decide how you would like to break it down? • Tasks • Diagnosis/Investigation/Treatment/Counseling • Context of care • ED/Inpatient/Outpatient • Competencies 24
- 25. How do I blueprint for my assessment? Content No. of items Diagnosis Investigation Treatment CAP 11 4 4 3 HAP 3.65 1-2 0 1-2 Otitis media 3.65 1-2 0 1-2 UTI 7.5 2 3 2 Meningitis 3.65 1 1 1-2 Endocarditis 3.65 1 1-2 1 Osteo/Septic arthritis 2.45 1 1 0 Septicemia 7.5 2-3 1-2 3 Cellulitis/Skin infection 4.9 2-3 0 2 HIV 2.45 1 1-2 0 25
- 26. How do I blueprint for my assessment? Final blueprint often looks like a matrix • Patient group X Task • Clinical presentation X Context of Care • Clinical Activities X Competencies 26
- 27. Exercise #1 • Using worksheet #1 start working on a blueprint for your purpose • Get together with colleagues and choose one blueprint to work on • Large group discussion 27
- 28. Microsoft clip-art, 2012 28
- 29. Assessment Tool Boxes What will I use to assess my blueprint? 29
- 30. How do I assess what I want to? Blueprint • Establishes what you wish to test Assessment tool box • Helps to determine how you will test 30
- 31. Assessing the 7 C’s 31
- 32. Assessing the Competencies Professional Competence • More than a demonstration of isolated competence Eraut, 1994 • “…when we see the whole, we see its parts differently than when we see them in isolation” Polanyi, 1969 32
- 33. Assessing the Competencies • Context specific • What may be considered “competent” in one context may not be in a different context. 33
- 34. Assessment tools for Competencies RCPSC suggest: • In-training evaluations (ITERs) • MCQs • SAQs • Essays • Oral examinations • OSCEs • Simulations • Log books Microsoft clip-art, 2012 34
- 35. CanMEDS Evaluations and PGE Med expert • ITER>Oral>MCQ>SAQ Communicator • ITER>Oral>OSCE>sim Other roles • ITER>Oral>OSCE Chou et al. Medical Education, 2008 35
- 36. CanMEDS Evaluation and PGE PG directors levels of satisfaction with CanMEDs evaluations • Medical Expert: • satisfied to very satisfied • Communicator, collaborator, scholar, and professional • Neutral to satisfied • Manager and health advocate • Dissatisfied to neutral Chou et al. Medical Education, 2008 36
- 37. ACGME Competencies • Patient care • Medical knowledge • Practice-based learning and improvement • Inter-personal and communication skills • Professionalism • Systems-based practice 37
- 38. Assessment tools for ACGME Competencies ACGME tool box: Record review Chart stim. Recall Checklist Global rating SP OSCE Simulations 3600 global rating Portfolios MCQs Oral exam Log books Patient survey 38
- 39. Assessment Challenges when Charting the 7 C’s • Socially negotiated educational competencies • Accreditation agencies mandate that competencies be assessed • Tools don’t match single competencies independently except for medical knowledge/expert • Other competencies/roles • Reflect personal attributes? • What is desirable VS what is observable? Lurie, Medical Education, 2012 39
- 40. Potential solutions? • Linking abstract competencies to observable behaviors • Key competencies/Enabling competencies • Mapping competencies to observable behaviors • Milestones, Entrustable Professional Activities (EPAs) 40
- 41. Word of caution “…a tendency to describe general competencies in exhaustive detail, leading to bulky, fragmented documents that lose practical value for education as they become less and less connected with the real world.” ten Cate et al. Medical Teacher 2010 41
- 42. Building your own assessment toolbox • Define the purpose of your assessment • Define the population you are assessing • From your blueprint, what is best assessed with what tool • Is it going to be reliable? Feasible? Acceptable? • Will the result interpretation be valid? 42
- 43. Exercise #2 • Create your own toolbox using worksheet #2 • Discuss your toolbox with a colleague • Group discussion • Challenges? 43
- 44. Microsoft clip-art, 2012 44
- 45. In Summary • Define what you wish to assess by designing a blueprint • Decide how you will assess • Understand the challenges and limitations to assessing the 7 C’s • Stay tuned to further developments 45
- 46. Thank You! Questions? Please give me feedback at ctouchie@mcc.ca 46
- 47. Useful references 1. Epstein RM and Hundert EM. Defining and assessing professional competence. JAMA 2002;287:226-235 2. Crossley J, Humphris G, Jolly B. Assessing health professionals. Medical Education 2002;36:800-804 3. Lurie SJ. History and practice of competency-based assessment. Medical Education 2012;46:49-57 4. Coderre S, Woloschuk W, McLaughlin K. Twelve tips to blueprinting. Medical Teacher 2009;31:322-324 47