Assessment Consistency: Is Everyone on the Same Page?
1 like264 views
The document discusses the importance of consistency in exam creation and assessment within the Department of Pharmaceutical Sciences at St. John Fisher College. It outlines faculty development workshops aimed at improving exam quality through clear learning outcomes, peer review, and standardized exam formats. The emphasis is on creating a validated test bank and guidelines to enhance the clarity and effectiveness of assessments.
Assessment Consistency: Is Everyone on the Same Page?
- 2. Participants will learn: 1. How to use faculty development workshops to provide the foundation of writing quality exams. 2. How to effectively use the peer-review process. 3. How to create consensus and guidelines for creating and launching exams.
- 3. Why is consistency important? • Consistency contributes to exam quality • Exams should reflect content learning • Reduces student confusion or frustration • Exams are high-stakes • Data are used for many purposes
- 4. Faculty Development for Consistency • To prepare faculty for writing high quality exams, we conducted a series of workshops: – learning outcomes – exam blueprints – coding to Bloom’s Taxonomy – peer-review process • Each of these workshops helped to form the foundation of constructing exams which truly reflected student learning.
- 6. Clearly written outcomes and exam maps • We made the decision to use ExamSoft® as our tool for course assessments. • Before any exams were created using the technology we had several faculty development workshops. • Development sessions led by NABP – Revision of course learning outcomes by faculty in the course – Techniques for writing good exam questions – Use of exam Maps – Proper coding to Blooms Taxonomy • All faculty are writing and coding their own exam questions.
- 7. Exam map: is my exam reflective of what I taught? Course learning outcome: Based on the site and mechanism of action, predict the therapeutic and side effects associated with pharmacological agents used to treat various disease states. Course Activity Lecture LO Knowledge Application Synthesis Lecture 2 (9.11.12 – 9.13.12) APPE lectures Lecture, TP, patient scenario EPO REMS · pg. ; Bank question 1420 (MC) Anemia Outcome 4 1 Iron + Vitamin C · pg. 9; Bank question 1525 (FITB) Anemia Outcome 4 1 Anemia from RBC destruction • pg. ; Bank question 1527 (SA) Anemia Outcome 1, 2, 3 1
- 8. Continued updates and quick tips • Use time during faculty retreats and meetings to practice coding sample questions, to address coding inconsistencies, holes in our curricular coverage, questions/concerns • Email reminders with “how to” – Ex. How to write a good FITB question • Binders for all faculty with sample materials – Appendix B topics, WSOP program outcomes, Bloom’s Taxonomy verbs
- 10. The peer review process • “Peer-reviewed articles provide a trusted form of scientific communication.” • “Even if you are unfamiliar with the topic or the scientists who authored a particular study, you can trust peer- reviewed work to meet certain standardsof scientific quality.” Diagram and quotes from: http://undsci.berkeley.edu/article/howscienceworks_16
- 12. Our Process § This is a voluntary process for faculty who want to participate § 2-3 reviewers/exam § Faculty are invested in the process § Reviewers DO NOT make changes to exams, and item writers can choose not to make suggested changes § Student data is much more reflective of what was learned versus could my test item be deciphered
- 13. Quality of questions: Peer reviewed test item YES NO Comments/ Suggestions 1 Is the question clear to the reader? 2. Are there typographical errors? 3. Is the question coded to the appropriate level of Bloom’s Taxonomy? 4. Is the question coded appropriately to the learning outcome(s)? 5. If the question has a history, did it perform well in the past? 6. Is the content level appropriate for the course? 7. Other considerations: “positive” language, jargon free, limited abbreviations, double-jeopardy, exam time
- 15. How to learn from others • Problem areas: 1. FITB questions 2. Short answer/essay (character limits) 3. Eliminating K-type 4. Select all that apply
- 17. Creating Formatting Guidelines for Consistency • Ensures a consistent look to all exams and exam reports • Consider having a “Quick Tips” document that outlines these for all faculty • Guidelines – Question format – Exam creation and postings – Exam administration – Exam reports
- 22. Consistency in Exam Creation • Prevent logistical errors in exam with a checklist: – Correct # of points – Check question groupings – Override font and font size – Check settings on partial credit, select-all-that-apply and k-type question – Export copy of exam for last-minute spelling errors – Preview exams before posting
- 26. Consistency in Exam Administration • Exam details • Assessment details • Policies • Contact information
- 28. Conclusions • Good exam consistency involves question writing, exam creation, exam posting, exam administration, and exam reports. • Training is required to get all faculty on the same page, and to emphasize the importance of consistency. • A “Quick-Tips” guide is useful for logistical recommendations for consistency.