Online Course Assessment Part 1
7 likes2,116 views
The document summarizes key points from a presentation on designing online course assessments. It discusses foundations of online assessment including validity, reliability, and alignment. It also covers developing assessments, such as specifying objectives, selecting appropriate assessment types, and ensuring alignment between objectives, activities and assessments. Finally, it addresses creating an assessment toolkit, including choosing appropriate tools, criteria, and ensuring privacy compliance.
![What other assessment principles have you applied? [respond in chat]](https://image.slidesharecdn.com/1onlinestrategies-100316124143-phpapp01/85/Online-Course-Assessment-Part-1-17-320.jpg)
![Ideas for formatively assessing learning… [respond in chat]](https://image.slidesharecdn.com/1onlinestrategies-100316124143-phpapp01/85/Online-Course-Assessment-Part-1-30-320.jpg)
![Where do you start when you are developing an assessment? [respond in chat]](https://image.slidesharecdn.com/1onlinestrategies-100316124143-phpapp01/85/Online-Course-Assessment-Part-1-39-320.jpg)
![What portion of your assessments do you design yourself? [respond in chat]](https://image.slidesharecdn.com/1onlinestrategies-100316124143-phpapp01/85/Online-Course-Assessment-Part-1-40-320.jpg)
![Q and A [respond in chat]](https://image.slidesharecdn.com/1onlinestrategies-100316124143-phpapp01/85/Online-Course-Assessment-Part-1-57-320.jpg)
![FERPA and Web 2.0 (handout) This [insert name of technology tool] is for academic use of [insert name of course], [insert term/year]. It is open to the public for the purpose of sharing our work with the larger Internet community. To use this [insert name of technology tool] responsibly please observe all laws and university policy that are incorporated into the Codes of Conduct and Academic Integrity. Some specific aspects of law and policy that might be well to remember are prohibitions against copyright infringement, plagiarism, harassment or interference with the underlying technical code of this software. As a student using this [insert name of technology tool], certain rights accrue to you. Any original work that you make tangible belongs to you as a matter of copyright law. You also have a right to the privacy of your educational records as a matter of federal law. Your contributions to this [insert name of technology tool] constitute an educational record. By contributing to this [insert name of technology tool], and not taking other options available to you in this course equivalent to this assignment that would not be posted publicly on the Internet and not available for the editing by others, you consent to the collaborative use of this material as well as to the disclosure of it in this course and potentially for the use of future courses.](https://image.slidesharecdn.com/1onlinestrategies-100316124143-phpapp01/85/Online-Course-Assessment-Part-1-79-320.jpg)
![Thank you! [email_address] [email_address]](https://image.slidesharecdn.com/1onlinestrategies-100316124143-phpapp01/85/Online-Course-Assessment-Part-1-82-320.jpg)
Online Course Assessment Part 1
- 1. Designing and Developing Online Course Assessments Day 1: Online Strategies for Assessment Design Dr. Veronica Diaz, Maricopa Community Colleges Dr. Patricia McGee, The University of Texas at San Antonio
- 2. Day 1 Agenda Welcome Part 1: What are the foundations of online course assessment? Part 2: What is the process of developing assessment? Part 3: What should go into an Assessment Toolkit? Q&A
- 3. What are the foundations of online course assessment?
- 4. POLL: Which best describes your experience with assessments? Developed my own assessments Used publisher-created assessments Worked with others to create assessments Used assessments developed by someone other than publisher
- 7. Principles of Sound Assessment
- 8. Focus on …. The learner Quality and relevant feedback
- 9. Assessment Principles for Effective Design Valid and Reliable Actionable Efficient and Cost-effective Engaging and Interesting Source: Allen, M. J., October 2007
- 10. Online Assessment Principles
- 11. Align objectives – activities – assignments – assessments Objective Activity Assignment Criteria Given everyday objects, the learner will identify three out of four objects with masses relatively equivalent to 1 cubic foot of water. (1) View Water flow and force. (2) Complete water displacement simulation Read CH 3 Complete CH 3 worksheet (1) Complete 5 out of 6 challenge problems correctly and turned in on time
- 12. Provide multiple and alternative modes of assessments
- 13. Integrate strategies to prevent cheating Randomized objective quizzes/tests Timed and authenticated Multiple ways to demonstrate understanding Choices Duplication Repetition Revise and resubmit Small group assessments for accountability
- 14. Provide accurate and timely feedback for performance assessments Detailed expectations, e.g., rubrics Respond within a pre-designated timeframe Peer critique review before final submission
- 15. Evaluate your assessment practices regularly Are their persistent problems? Is there an area of the course that gets “stuck” and remediation or review is needed? Is a lot of time spent on answering questions or clarifying?
- 16. Online mid-semester and/or end-of-semester course evaluations Does the student know what they are accomplishing? Are their “invisible” factors that are impacting assessment?
- 17. What other assessment principles have you applied? [respond in chat]
- 18. Classroom vs. Online Assessment
- 19. Face-to-Face vs. Online Campus Classroom One to many Just in time assessment Immediacy Focus on methods
- 20. One to many Just in time assessment Immediacy Focus on methods One to one Just in need assessment Ambiguity Focus on strategies
- 21. Differences? Time Security Integrity Resources Learner autonomy Lack of just in need responses Getting, keeping, and directing attention Clarity of instructions and expectations
- 23. Transfer of learning Objectives? Outcomes? Monitor and Adjust What Students Know transfer of learning What students can do Facts Procedures Principles Strategies Critical thinking Problem finding/solving Creative thinking Typically measured by objective assessments Typically measured by performance assessments Homework, quizzes, tests, exams Projects, cases, problems, designs, experiments
- 24. Progressive Assessment: Instructor POV (from Gardner, 2006, p. 105 adapted from Harlen, 2000)
- 25. Progressive Assessment: Series of Events (from Gardner, 2006, p. 105 adapted from Harlen, 2000)
- 28. Formative Assessment Informs the learner of what they have learned Informs the instructor of next steps Can be informal or formal
- 29. Formative: What to keep in mind What kinds of issues are involved? How does the learner know where they are on their progression towards competency and mastery? How can we document/illustrate/measure? Has the learner practiced? Other?
- 30. Ideas for formatively assessing learning… [respond in chat]
- 31. Summative Assessment Documents completion & achievement Provides barometer against benchmark Formal
- 32. Summative: What to keep in mind What kinds of issues are involved? Where do benchmarks come from? How do we illustrate progress against benchmarks? Other?
- 38. WHAT IS THE PROCESS OF DEVELOPING ASSESSMENT?
- 39. Where do you start when you are developing an assessment? [respond in chat]
- 40. What portion of your assessments do you design yourself? [respond in chat]
- 41. POLL: Select the 2 most common assessments you’ve used Multiple choice tests Written exams Problem-based projects Research papers Case studies Short answer Large projects Essay questions
- 42. Determining what needs to be assessed
- 43. Assessment Development Specify educational objectives/intended outcomes. Select assessment measures and techniques. Specify assessment criteria. Evaluate student performance on exams/projects for course grades. Evaluate student performance on course measures to assess effectiveness.
- 44. Role of Objectives Review course objectives Ask, are these for skills, knowledge, or abilities Are they measurable Demonstrate (authentic) or Reflect on knowledge/experience What do they say about student outcomes What is the goal of assessment, what do you want the student to be able to do or demonstrate they can do Use verbs to describe this Share with students
- 45. Objective ABCDs Audience (the learners) Identify who it is that will be doing the performance (not the instructor). Behavior (Performance): Make sure it is something that can be seen or heard. Condition (under which the learners must demonstrate their mastery of the objective): What will the learners be allowed to use? What won't the learners be allowed to use? Degree (HOW WELL the behavior must be done): Common degrees include: Speed, Accuracy, Quality
- 46. Objectives Activity Write a research report with abstract, introduction, procedure, results, discussion, conclusions, and recommendations. You will learn to analyze and interpret data. You will learn about about theoretical issues underlying communicative and task-based language teaching. Describe the political and social factors impacting the implementation of biotechnology in agriculture.
- 47. Alignment of course activities & outcomes
- 48. Assessment and Learning
- 49. Example: recall of factual knowledge Use multiple choice or short response assessment Example: complex learning outcomes, such as reasoning, communication, teamwork a performance assessment is likely to be appropriate
- 50. What needs to be assessed? Course goals Module objectives Activity objectives Content Knowledge: declarative, intellectual skills (concept, principle, procedure, problem solving) Thinking Skills Inter/Intrapersonal Skills: collaboration, cooperation Technology Literacy/Proficiency Other
- 51. Performance vs Objective Assessments Via another activity Cases Projects Presentations Debates/negotiations Research paper Be the activity itself Quiz Multiple-choice test Simulation Learning can be assessed…
- 52. Do your… Learning objectives Resource and materials Learner interaction and activities Course technologies Support student performance?
- 53. Assessment Alignment Objective is to “design a research study related to a professional field,” but the assessment is a short answer test Objective is to “understand particular statistical methods for various types of research,” but the assessment is to write an essay
- 55. Design vs Selection Design High stakes assessments Customized for your course Process-oriented Cases PBL Projects Classroom assessments (CATs) for an online environment Select Self assessments Short, quick checkpoints Repeatable with large question bank Low stakes
- 56. Bo sure to… Be explicit in directions Don’t get off task Use checklists Provide examples
- 57. Q and A [respond in chat]
- 58. WHAT SHOULD GO INTO AN ASSESSMENT TOOLKIT?
- 59. P. McGee
- 60. Determining an appropriate strategy For learning outcomes
- 61. Learners should know…. That an assessment is coming A word about helping students plan What happens after the assessment How to prepare for an assessment What participating in the assessment will look like Practice
- 62. Tools Assessments Technologies Methodology or learning framework Deploying the method or framework Disciplinary considerations and requirements
- 63. Technology and the Cycle
- 64. Direct Assessments Classroom Assessment Techniques Multiple choice tests Embedded Test Questions Pre and Post Tests Discussion Threads
- 65. Indirect Assessments Exit interviews or student self-report surveys Final course grade Tracking of student data Focus groups
- 66. Assessment Considerations Provides guidance to further develop conceptual framework Continuous process (formative) Guides the student to mastery of the learning outcomes Assessment strategy becomes foundation for developing the instructional design of the online course
- 67. Suggestions for Better Assessment Build multiple “check point” assessments Allow open note/book timed tests Have in class quizzes and tests as well Project-based assessments Portfolio assessments Assign group work in a wiki area that tracks student participation Provide best practice guidelines for taking exams online (See handouts)
- 68. Bloom’s Taxonomy Focus on learner Focus on measure of learning
- 69. Bloom’s & Web 2.0 Processes Tools Attributes Remember Recognizing, recalling Visual/Text/Audio stimuli, selecting, feedback Understand Interpreting, classifying, comparing, summarizing, explaining Sorting, tagging, labeling, entering, selecting Apply Executing, implementing Manipulating, entering, feedback Analyze Differentiating, organizing, attributing Selecting, grouping, altering, tagging, labeling Evaluate Checking, critiquing Commenting, entering, responding Create Generating, planning, producing Adding, generating, combining, publishing
- 71. Assessment criteria should be… Measureable Align with outcomes Achievable within online format Documentable Clear, articulated expectations Does not require additional technology skills
- 72. For example Chat – content learning Answer questions correctly Ask questions that Helped others learn Extended discussion Make contributions to discussion that extended materials Chat - performance Participate throughout discussion Do not interrupt Respond when called upon When appropriate, come to chat prepared.
- 73. Outcome + Criteria = Assessment How are you communicating that to learners? How will you document the learning ? Do you expect learner to learner a new tool for assessment ? Have you provided for practice before assessment?
- 75. Course Management Systems Privacy and security Familiarity Integration Support Limited tool options Some web 2.0 options
- 76. CMS to Web 2.0 and Assessment http://web20-toolkit.wetpaint.com/ http://elearningtools.wetpaint.com/?t=anon http://maricopatech.wetpaint.com/ http://web2educationuk.wetpaint.com/
- 77. Web 2.0 Considerations Cost Accounts Technical requirements Is the tool generative? If so, how will product generated by learner be accessed by others? Exportable product Technical skills required Support Can instructor embed or attach instructions within tool? Shared/selective viewing Public viewing
- 78. Web 2.0 Considerations Tools not necessarily developed for an educational audience No obligation to users Ever-changing Require separate logins/accounts/fragmentation No centralized institutional support (usually) Reliance on internet connection (high speed) Lack of security Learning curve Variety of use and selection of tools could overwhelm students; lack of a common experience across courses
- 79. FERPA and Web 2.0 (handout) This [insert name of technology tool] is for academic use of [insert name of course], [insert term/year]. It is open to the public for the purpose of sharing our work with the larger Internet community. To use this [insert name of technology tool] responsibly please observe all laws and university policy that are incorporated into the Codes of Conduct and Academic Integrity. Some specific aspects of law and policy that might be well to remember are prohibitions against copyright infringement, plagiarism, harassment or interference with the underlying technical code of this software. As a student using this [insert name of technology tool], certain rights accrue to you. Any original work that you make tangible belongs to you as a matter of copyright law. You also have a right to the privacy of your educational records as a matter of federal law. Your contributions to this [insert name of technology tool] constitute an educational record. By contributing to this [insert name of technology tool], and not taking other options available to you in this course equivalent to this assignment that would not be posted publicly on the Internet and not available for the editing by others, you consent to the collaborative use of this material as well as to the disclosure of it in this course and potentially for the use of future courses.
- 80. Q&A
- 81. Take-Aways Keep the learner in the forefront Focus on timely feedback and benchmarks Align objectives – activities - assessments
- 82. Thank you! [email_address] [email_address]
Editor's Notes
- #3: Veronica – Welcome and Agenda
- #4: Patricia
- #6: Patricia – Assessment is aligned with evaluation process but serve different purposes. While assessment measures the achievement and progress of the individual, evaluation indicates the successful implementation of a course of instruction and how learning is transferred to other contexts.
- #7: Patricia – Assessments are conducted for a variety of purposes. In an online environment these are particularly crucial since evidence may not be readily available or evident to the course instructor.
- #9: Patricia
- #10: Patricia - Valid: directly reflects the learning outcome being assessed Reliable: inter-rater reliability when subjective judgments are made Actionable: results help faculty identify what students are learning well and what requires more attention Efficient and cost-effective in time and money Engaging to respondents, so they’ll demonstrate the extent of their learning Interesting to stakeholders, care about results and are willing to act on them Triangulation: multiple lines of evidence point to the same conclusion
- #12: Patricia Match learning objectives with assessments Take precautions to limit the possibility of cheating Communicate assessment tasks clearly Use formative assessment to promote deeper learning; consider alternative forms of assessment such as portfolios Use self-assessments to improve learning and self-awareness Have students conduct peer-assessments (may be particularly effective when used in conjunction with group work) Multiple sources including http://www.sjc.cc.nm.us/pages/2852.asp
- #13: Patricia Match learning objectives with assessments Take precautions to limit the possibility of cheating Communicate assessment tasks clearly Use formative assessment to promote deeper learning; consider alternative forms of assessment such as portfolios Use self-assessments to improve learning and self-awareness Have students conduct peer-assessments (may be particularly effective when used in conjunction with group work) Multiple sources including http://www.sjc.cc.nm.us/pages/2852.asp
- #14: Patricia
- #15: Patricia Performance feedback is challenging in an online environment because performance must be re-defined. Whether performance involved actions and productions that can be easily relayed online, feedback is key to the learner understanding “how they did.” Depending upon the type of course and the numbers of students there are a variety of strategies to provide feedback.
- #16: Patricia
- #17: Patricia Evaluations can inform you if the assessment are helping students understand what they are accomplishing. They can also identify other barriers or ‘invisible’ problems.
- #18: Patricia
- #19: Patricia
- #20: Patricia -When considering differences between face-to-face and online contexts for learning, several factors are in critical when designing, developing, and implementing online assessments. In a campus classroom the instructor is typically delivering instruction to a group of students, while similar in an online classroom, the students are separated and are realistically in a classroom on one. Students can ‘feel’ that instruction is one-on-one and the instructor is much more in tune with what they are experiencing. Students can rely on the instructor to keep track of everything they do, while in a classroom the instructor is more group focused. In a classroom assessments are scheduled whereas in an online classroom, assessments may be needed if there are disparate responses, lack of achievement, etc. In a classroom the instructor knows and seeks out ways of assessing student understanding in real time through multiple modalities and strategies. In a.n online class there is much ambiguity about understanding. In a classroom courses are typically designed around teaching methods: lecture, presentation, research, etc. In an online course, content is typically chunked into segments where the focus is more strategy-based: discussions, chats, group work, etc.
- #21: When considering differences between face-to-face and online contexts for learning, several factors are in critical when designing, developing, and implementing online assessments. In a campus classroom the instructor is typically delivering instruction to a group of students, while similar in an online classroom, the students are separated and are realistically in a classroom on one. Students can ‘feel’ that instruction is one-on-one and the instructor is much more in tune with what they are experiencing. Students can rely on the instructor to keep track of everything they do, while in a classroom the instructor is more group focused. In a classroom assessments are scheduled whereas in an online classroom, assessments may be needed if there are disparate responses, lack of achievement, etc. In a classroom the instructor knows and seeks out ways of assessing student understanding in real time through multiple modalities and strategies. In a.n online class there is much ambiguity about understanding. In a classroom courses are typically designed around teaching methods: lecture, presentation, research, etc. In an online course, content is typically chunked into segments where the focus is more strategy-based: discussions, chats, group work, etc.
- #22: Patricia
- #23: Patricia
- #24: Patricia - In thinking about assessment we need to start by thinking about what we expect students to know. Undergraduate students particularly are focused on the acquisition of foundational knowledge that are typically measured by objective assessments which work fairly well in an online environment. If a course is designed to have the learner transfer what they have learned into application, then performance measures and assessments must be designed to allow them authentic ways to perform. In a traditional classroom, I would argue, performance assessments are easier to implement because you have control over the environment and variables, and you can see the processes students use. In an online course, these processes are often invisible and if students are novices to higher education or distance learning, then their lack of sophistication and expertise may present challenges.
- #25: Patricia – progressive assessment is important in online contexts because the student may not always be aware of where they stand in accomplishing course objectives. In a ftf classroom, teachers typically view progressive assessment as a series of learning events, collection and interpretation of evidence judged against pre-determined criteria, culminating in a report. References Caldwell, B. & Carter, E. M. A. (1993). The return of the mentor: Strategies for workplace learning . Routledge Gardner, J. (2006). Assessment and learning. SAGE.
- #26: Patricia- Progressive assessment can also be viewed as a series of events. Here we may start with goals from which student either provide or generate evidence before or during a class. Then students and instructor determine criteria are used to judge achievement. The outcome of this judgment results in next steps for learning References Caldwell, B. & Carter, E. M. A. (1993). The return of the mentor: Strategies for workplace learning . Routledge Gardner, J. (2006). Assessment and learning. SAGE. .
- #27: Patricia – in an online environment, progressive assessment in learner-centered; learners keep track, participant in, discuss, and communicate their understanding, mastery, deficits, etc. with each other and the instructor.
- #28: Patricia – In an online environment, progressive assessment allows the learner to document and recognize their achievement of learning objectives
- #29: Patricia
- #30: Patricia
- #31: Patricia
- #32: Patricia
- #33: Patricia
- #34: Patricia - The relationship between formative and summative assessment can be viewed in several ways. Some course designs support a semester long progression of assignments the lead to a final, summative product, project, or final evaluative event – an exam, performance, presentation, etc.. In this model, formative assessments determine whether the learner has satisfactory met the criteria and is prepared to proceed.
- #35: Patricia - An example of such an approach is a writing or research-based course.
- #36: Patricia - Another approach is to have periodic assessment that measure student learning but do not indicate a progression towards a final summative event. In this design students can proceed through the materials once they have completed the assessment.
- #37: Patricia - One example of this approach in an online course design is where students take different approached to the assignment and proceed through activities without intervention if they haven’t mastered content.
- #38: Patricia
- #39: Veronica
- #40: Veronica
- #41: Do in chat
- #42: Veronica
- #44: Veronica
- #45: Veronica - Note: if objectives are poorly written, the assessment will be difficult to construct
- #46: Veronica
- #47: Veronica - Let’s go through each and in the chat make suggestions on how it could be improved, then suggest an an assessment tool that could be used to assess
- #49: Veronica
- #50: Veronica
- #51: Veronica
- #52: Veronica
- #53: Veronica - Alignment
- #54: Veronica
- #55: Veronica
- #56: Veronica
- #57: Veronica
- #58: Veronica
- #59: Patricia and Veronica
- #60: Patricia
- #61: Veronica
- #62: Veronica
- #63: Veronica
- #64: Veronica
- #65: Veronica
- #66: Veronica- Students or others report their perception of how well a given learning outcome has been achieved
- #67: Veronica
- #68: Veronica
- #69: Veronica
- #70: Patricia - handout
- #71: Patricia – Selecting criteria is important for either framework
- #72: Patricia – Selecting criteria is important for either framework
- #73: Patricia
- #74: Patricia Look at the objective (s) Think about the criteria for assessment
- #75: Veronica
- #76: Veronica
- #77: Veronica
- #78: Veronica See handout
- #79: Veronica
- #80: Veronica- See handout
- #81: Veronica and Patricia