Evaluating the Quality of Online Teaching
Download as PPTX, PDF5 likes7,998 views
The document evaluates the quality of online teaching, focusing on definitions of quality in education and digital formats, key components for assessing e-learning, and faculty evaluation systems. It highlights the importance of continuous improvement, accreditation standards, and various evaluation methods including student feedback, self-assessments, and peer reviews. The document also discusses adaptations for online evaluations, practical examples, and a rubric for assessing online course quality.
Evaluating the Quality of Online Teaching
- 1. Evaluating the Quality of Online Teaching Barry Dahl, Sr. Community Manager, D2L
- 2. What does “Quality” mean?
- 3. What is a Good Definition for Quality Generally speaking… •High Grade •Superiority •Excellence
- 4. Define Quality in Education Generally speaking… •High standards •Consistently applied •With an eye on efficiency (or value for money)
- 5. Define Quality in Online/Digital Education Generally speaking… Passing the Quality Matters (QM) rubric review But… It shouldn’t be
- 6. 3 Major Components of e-Quality Learning Teaching Design
- 7. Learning Level Is High Teaching Level Is High Course Design Meets Standards Learning Assessment Measuring the Major Components
- 8. Comprehensive Faculty Evaluations • Purpose • Accreditation • History of Online Faculty Evaluation • Components • A few examples • What about classroom observations for online courses?
- 9. Purposes of Faculty Evaluation 1. Encourage excellence in teaching and learning. 2. Facilitate long-term quality improvement by continually monitoring instructional performance. 3. Provide constructive feedback to faculty by identifying areas of strength and areas for improvement in classroom instruction. 4. Inspire professional growth and development.
- 10. What about Accreditation? • Here’s an example from the Higher Learning Commission • Guidelines for the Evaluation of Distance Education (On-line Learning) • NOTE: most other regional accreditors use the same or similar language.
- 11. Guidelines for the Evaluation of DE Faculty responsible for delivering the on-line learning curricula and evaluating the students’ success in achieving the on-line learning goals are appropriately qualified and effectively supported. • Examples of evidence: a. On-line learning faculties are carefully selected, appropriately trained, frequently evaluated, and are marked by an acceptable level of turnover.
- 12. A Little History…Maybe • The first known (to me, anyway) online faculty evaluation system was started in 2001 at Park University. • Initially based on the evaluation system used for face-to-face classroom instructors. • As such, they attempted to replicate the traditional classroom evaluation in the online classroom.
- 13. But Online Was (Is) Different The unique features and functions in an online classroom were not initially considered. For example, the original evaluation system did not include an examination • of learning outcomes • of overall classroom management • facilitation and guidance • the concept of faculty presence • communication response rates • accessibility of materials • course-related administrative tasks, etc.
- 14. Adapting the Instrument • To address the unique evaluation concerns for online teaching, Park University College for Distance Learning developed a formalized Online Instructor Evaluation System (OIES). • OIES launched as a pilot in Fall 2004, and was used through 2008. • OIES morphed into the Faculty Online Observation (FOO) model and is still in use today.
- 15. Evaluation System Components • Student Evaluation of Instruction • Faculty Self-Evaluation • Professional Development Plan • Faculty Peer Review • Faculty Portfolio • Supervisor’s Evaluation of Faculty Usually some combination of the above components
- 16. Student Evaluation of Online Instruction • Not our main focus here. • However, some questions or issues to focus on: • How to get decent response rates • Use incentives, or not? • What effect does the length of survey have on responses? • Does it need to be comparable to F2F evaluation surveys? • How valuable are these overall?
- 17. Faculty Self-Evaluation • You might love these, you might hate these. You also might mostly ignore these. • Let’s look at two examples: • Open-ended questions • Specific questions tied to student eval
- 18. Open-ended questions Sample questions from faculty self-evals: 1. How has your understanding of your role as a faculty member changed/developed since your last evaluation? 2. As you prepare for future semesters, what more could you do to provide students with a successful learning experience? 3. What more can the university do to support you in your professional goals and development?
- 19. Questions tied to student evaluations Wharton County Junior College • Online Course Evaluation (completed by students) 4. Instructor requires me to be an active participant in class. 9. Instructor is present regularly in the course (announcements, emails, discussion board, feedback, etc.). • Online Faculty Self-Evaluation 4. I require students to be an active participant in class. 9. I am regularly present in my course (announcements, emails, discussion board, feedback, etc.).
- 20. Faculty Peer Review of Online Teaching Great example comes from Penn State University • College of Earth and Mineral Sciences • Based on the “Seven Principles for Good Practice in Undergraduate Education” • Two Parts: • An Instructor Input Form to be completed for the reviewer by the reviewee in advance of the peer review, and • The actual Peer Review Guide for Online Teaching at Penn State, which is to be completed by the reviewer during the peer review.
- 21. Screenshot of Guide Last revision June 23, 2017 - Ann H. Taylor, Dutton e-Education Institute, College of Earth and Mineral Sciences, The Pennsylvania State University
- 22. Examples of Evidence to Look For Last revision June 23, 2017 - Ann H. Taylor, Dutton e-Education Institute, College of Earth and Mineral Sciences, The Pennsylvania State University
- 23. Where to Look, and Resources Links Last revision June 23, 2017 - Ann H. Taylor, Dutton e-Education Institute, College of Earth and Mineral Sciences, The Pennsylvania State University
- 24. Portfolio for Faculty (Austin CC) The portfolio will include the following: a. Syllabus for each course (not section) taught, up to four courses b. Samples of major assignments, tests, projects c. Statement of Teaching Philosophy d. Course Commentary e. Professional Development Plan
- 25. Supervisor’s Evaluation of Teaching • Most schools have some sort of process where a Dean, Department Head, Director, or similarly positioned person evaluates faculty performance. • One part of this process is typically the classroom observation. • In-class observations for online courses/instructors are inherently different from F2F class observations.
- 26. Issues with Teaching Observations Timeline: • F2F – typically a single class period • Online – as much or as little as you choose/allow Familiarity: • Most deans have taught F2F at one time or another • Most deans have NOT taught online, although this is improving Criteria: • F2F – Most schools have established criteria for observations • Online – Many do not have established criteria
- 27. Remaining Agenda Review the Online Faculty Evaluation Rubric • Sharable & Editable (Creative Commons BY-NC-SA 3.0) Components included: 1. Pre-evaluation worksheet 2. Professional Development documentation 3. Expectations to be Evaluated • Examples of Meeting Expectation • Evaluator Comments
- 28. Creative Commons License • Creative Commons BY-NC-SA 3.0 • You are free to: • Share — copy and redistribute the material in any medium or format • Adapt — remix, transform, and build upon the material • The licensor cannot revoke these freedoms as long as you follow the license terms.
- 29. Under the following terms: Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use. NonCommercial — You may not use the material for commercial purposes. ShareAlike — If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original. No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.
- 30. Quality of Learning Quality of Instruction Quality of Course Design Learning Assessment Measuring the Major Components
- 31. Quality of Learning Quality of Instruction Quality of Course Design Learning Assessment Measuring the Major Components
- 32. Evaluation System Components • Student Evaluation of Instruction • Faculty Self-Evaluation • Professional Development Plan • Faculty Peer Review • Faculty Portfolio • Supervisor’s Evaluation of Faculty Usually some combination of the above components Shared rubric could be used for any of these pieces.
- 33. Poll Questions – show of hands Choose each statement that is true: 1. My school has established a process for online classroom observation to evaluate the quality of online teaching. 2. Our deans and/or other academic leaders have the online education experience and knowledge to effectively evaluate the quality of online teaching. 3. My school uses Quality Matters or a similar process/rubric for reviewing online course design. 4. My school has a formal process of faculty peer review of online course design. 5. My school has a formal process of faculty peer review for online teaching.
- 35. Pre-Evaluation Worksheet for Online Faculty Evaluation Faculty name Evaluator name Course to be evaluated Term/Semester Yes No Item: Have you previously taught this course in the online format? Have you previously had an administrative evaluation completed for this course, at this college? Are you the primary course designer for this course? If not, indicate designer's name below. Have you made significant design changes to this course during the past year? Additional comments:
- 36. Sources of Instructional Content “X” if Yes Item: (check all that apply) Traditional printed textbook – insert ISBN here: Textbook publisher's e-text or “e-pack” (add link if possible) Textbook publisher's website (add link to site) Independent e-book (add link if possible) Open Educational Resources (add link if possible) Multimedia components available on the Internet Instructor-owned instructional assets (describe below) College-owned instructional assets (describe below) Commercially-available instructional assets (CD-ROM, etc.- add link if possible) Instructor's or departmental website, blog, wiki, etc. (add links to sites) Other – indicate nature of other resources in comments section below. Additional comments:
- 37. Instructional Communication Tools Inside LMS Outside LMS No Item: (NOTE: Add URLS for any tools used outside course) Discussion forums Chat room Web conferencing tool Instant messaging or pager Assignments folder feedback comments Quiz feedback comments Email Blog or wiki Social networks or other websites Other – indicate nature of other tools in comments section below. Additional comments:
- 38. End of Pre-Eval Use the following space to communicate anything else that would be beneficial to you or the reviewer prior to the teaching evaluation. Adapted from: Lake Superior College Online Faculty Pre-evaluation Worksheet
- 39. Online Faculty Evaluation Rubric
- 40. Basic Info Professional Development: Is there documentation on file indicating the following? Faculty name Evaluator name Course evaluated Term/Semester Start/End dates Yes No Faculty participation in the following: Training sessions for faculty who are new to online teaching Online faculty peer review process Web accessibility training Online faculty workshops, webinars, conferences Additional comments:
- 41. Evaluation Grading Scale: A = Exceeds expectations B = Meets expectations C = Will meet expectations with minor changes D = Falls short of expectations
- 42. Category 1: Basic Expectations for Online Courses Grade Expectation Examples of Meeting Expectation Evaluator Comments 1A: Course syllabus information matches approved college curriculum. 1A: Syllabus will include, but not be limited to, these items: 1) Learning outcomes listed in course match official outcomes. 2) Course description matches official description. 3) Disabilities statement and other standard curriculum language are included. 1B: All required and optional classroom resources are clearly identified with information about access. 1B: Syllabus or introductory information should include: 1) textbook information, including ISBN 2) lab kits, or similar resources 3) equipment needed by students 4) software or other resources 1C: Course time line is clearly identified and easy to find; with due dates, and appropriate windows of opportunity. 1C: Students often choose online learning because they seek time flexibility in their schedule management. 1) The "window of opportunity" refers to the number of days that a student may complete an assignment/quiz/etc. A quiz period that starts on Monday and closes on Thursday is a four day window of opportunity. 2) Generally speaking, windows of opportunity from 3-7 days are reasonable. 3) Periodic due dates (for example, weekly), are important as many online students have been found to not keep up with coursework when time line is left wide open.
- 43. Category 1: Continued Grade Expectation Examples of Meeting Expectation Evaluator Comments 1D: Expectations for faculty response times are clearly articulated for all major course components. 1D: Response time, grading turnaround time, or feedback frequencies are clearly articulated to students for the following, if used in course: 1) email responses 2) quiz grading/feedback (continued next page) 3) assignment grading/feedback 4) discussion forum postings, etc. 1E: Instructor-prepared course materials are accurate, professional, and consistent. 1E: Expectations include: 1) Good grammar 2) No obvious errors, including typos 3) Language level is appropriate for the audience 4) Consistent look and feel on content pages and syllabus 5) “Instructor voice,” in text or multimedia, is appropriate. 1F: Instructor provides and maintains a welcoming, safe, and positive learning environment. 1F: Examples include: 1) Clearly state the types of behavior that will not be tolerated in the online class. 2) Post and enforce rules for netiquette. 3) Maintain a safe learning environment by: a) focusing on academic achievement, b) maintaining high standards, c) fostering positive relationships between students and faculty, d) reducing negativity, and e) resolving conflict
- 44. Category 2: Learning Experiences/Opportunities Grade Expectation Examples of Meeting Expectation Evaluator Comments 2A: Has designed learning opportunities that encourage higher order thinking skills such as analysis, evaluation, synthesis, problem solving, and critical thinking. 2A: Non-exclusive list of possibilities: 1) scenario-based problems 2) Case studies and/or simulations/games 3) Peer tutoring, 4 Journaling 5) Exercises in diagramming or visualizations 6) Cooperative learning in groups 2B: Instructor uses discussion forums effectively. 2B: Student discussions should serve to: 1) stimulate thinking, 2) challenge attitudes and beliefs 3) develop interpersonal skills 4) investigate questions without simple answers 2C: Learning opportunities are sufficient for students to achieve all of the expected course outcomes. 2C: Significant learning opportunities/requirements: 1) are correlated to specific course learning outcomes. 2) are not tangential to course learning outcomes.
- 45. Category 2: Continued Grade Expectation Examples of Meeting Expectation Evaluator Comments 2D: Learning experiences are current, relevant, and presented in a logical sequence. 2D: Smaller “chunks” of content or modules tend to enhance student learning. Content should be kept up-to-date (including links). When possible, course content in latter part of course should build on content from earlier parts. Possibilities for chunking include: by module, by week, by chapter, by topic 2E: Learning opportunities enable students to engage in a variety of activities that promote practice and a transfer of skills. 2E: For example: 1) Learners can sometimes choose among available options for activities that are relevant to them. 2) Variety of learning opportunities may allow students to engage with course content in more than one modality. Additional comments on Category 2:
- 46. Category 3: Activity and Interactivity Grade Expectation Examples of Meeting Expectation Evaluator Comments 3A: Class introductions and start-up procedures are appropriate for instructor and students. 3A: Getting off to a good start is crucial in eLearning: 1) Start building a sense of community among faculty and students during the 1st week. 2) Consider alternate strategies for course introductions (including audio/video) to build strong sense of instructor presence. 3) Create a quiz on key items from the syllabus and/or key services available to online students. 3B: Encourages students to be active participants in class. 3B: Examples include: 1) Require frequent contributions to the class by all students. 2) Prompt students to share their experiences and knowledge with other students. 3) Vigilantly track non-attendance or non-participation by lagging students. Contact students by email or send note to advisor for students not participating. 4) Add links to relevant and timely news articles and have students do the same.
- 47. Category 3: Continued Grade Expectation Examples of Meeting Expectation Evaluator Comments 3C: Instructor communicates with students in a clear and effective manner. 3C: Consider the following: 1) Be professional in your all communications. 2) Double check spelling and factual accuracy of news postings, discussion postings, emails, assignments, quizzes, etc. 3) Provide written feedback on quizzes/assignments beyond a simple “good job” or “needs work.” 4) Avoid jargon as much as possible. 3D: Instructor communicates with students in a timely manner. 3D: Possibilities 1) Challenge yourself to meet or exceed the feedback timing expectations that you have set for the class. 2) Take sanity breaks, but always let students know when they can expect to hear from you. 3) Intelligent agents (in D2L) can be effective ways of providing timely feedback on quizzes, dropbox assignments, etc.
- 48. Category 3: Continued again Grade Expectation Examples of Meeting Expectation Evaluator Comments 3E: Instructor has created multiple opportunities for student-to-student interactions. 3E: Through various online communication tools, require or encourage students to interact with others on a regular basis. Possible tools: 1) discussion boards 2) web conferencing 3) phone conferencing (or PC) 4) personal profile pages 5) email 6) social networking 3F: Students are required to engage in active learning exercises. 3F: Some examples include: 1) Create course activities that take the students away from the computer screen to accomplish certain tasks or achieve goals toward course outcomes. 2) Where appropriate, have students engage with community or industry members for course-related work. 3) Create assignments that encourage use of the college library, tutoring center, or other support services. Additional comments on Category 3:
- 49. Category 4: Evaluations, Assessments, and Grading Grade Expectation Examples of Meeting Expectation Evaluator Comments 4A: Grading policy is easy to locate, easy to understand, and complete. 4A: Instructor’s grading policy: 1) Should always be available in the course shell and in an easy-to-find location 2) Language used to describe grade determination should be easy to understand. 3) All components of the policy for determining student grades should be located in one place. 4B: Utilizes multiple assessment methods as required of all college courses. 4B: Consider a mixture of various options for graded work, including, but not limited to: 1) quizzes or exams, 2) writing assignments, 3) portfolios 4) lab reports, 5) discussion postings, 6) event participation and analysis, 7) problem sets, 8) oral presentations, 9) debates, 10) simulations, 11) case studies
- 50. Category 4: Continued Grade Expectation Examples of Meeting Expectation Evaluator Comments 4C: Utilizes the LMS grade book for students to track their progress through the course. 4C: The LMS grade book is essential since it provides a password- protected, one-stop shop for students to view all assigned grades. 4D: Take appropriate steps to mitigate security, honesty, and/or authentication concerns regarding student assessments. 4D: Examples might include: 1) Proctored exams 2) Timed quizzes/exams 3) Lock-down browser 4) Document submission via assignments folder 5) Plagiarism detection 6) Original writing assignments each term Additional comments on Category 4:
- 51. Category 5: Other Faculty Expectations 5A: Research Completed or in Progress Comments: 5B: Committee work or campus involvement Comments: 5C: Community service Comments: 5D: Service to students (non-teaching) Comments:
- 52. Finding the Rubric T&L Community Post Evaluating The Quality Of Online Teaching - Webinar Archive bit.ly/dahlrubric
- 53. Online Faculty Evaluation Rubric