Apt 501 chapter_7
- 1. Chapter 7 Developing Assessment Instruments Carolyn Jenkins-Haigler
- 2. Background An “assessment” is not just a test, but a range of items used to gauge a learner’s abilities and progress as well as quality of instruction in addition to evaluating the instructional mediums used. Criterion (or the plural version of criteria) is defined as: Criterion. A standard on which a judgment or decision may be based learners will a characterizing mark or trait answer 85% of quiz Criterion-referenced assessments, also known as objective questions reference assessments, focus on measuring performance items correctly to associated with learner performance and instructional integrity. demonstrate mastery of Benchmarks, or specific levels of reference, are used to gauge the lesson change in performance, attitudes, and other measurable items. It is important to promote learners to evaluate themselves – quality of work and performance “Self-evaluation and self-refinement are two of the main goals of all instruction since they can lead to independent learning.”
- 3. Objectives Describe the purpose for criterion-referenced tests. Describe how entry behaviors tests, pretests, practice tests, and posttest are used by instructional designers. Name four categories of criteria for developing criterion-referenced tests and list several considerations within each criterion category. Given a variety of objectives, write criterion-referenced, objective-style test items that meet quality criteria in all four categories. Develop instructions for product development, live performance, and attitude assessments, and develop a rubric for evaluating learners’ work. Evaluate instructional goals, subordinate skills, learner and context analyses, performance objectives, and criterion-referenced test items for congruence.
- 4. Types of Criterion-Referenced Tests Entry Skills Test Presented to learners before instruction is provided Assess prerequisite skills Learners may have harder time learning material if skills are lacking Pretest Presented to learners before instruction is provided Gauge learner mastery of material and help instructor to cater course material to meet needs of most learner Practice Test Presented during the instructional Used to facilitate learner participation during instruction Help to gauge learning and understanding Post Test Presented following the instructional Used to assess all objectives and skills from the instructional Help to evaluate instructional effectiveness and learner knowledge
- 5. Designing a Test Finding the best testing format to measure different areas/types of learning Verbal Information Domain ◦ HOW - Learners demonstrate understanding by remembering or not remembering information ◦ WHAT – Direct objective-style test items such as short-answer, alternative response, matching, and multiple-choice items Intellectual Skills Domain ◦ HOW – More difficult and complex to create – gauging knowledge of multi-faceted concepts ◦ WHAT – Objective-style test items, creation of a project/product, or performance/presentation Attitudinal Domain ◦ HOW – Also more difficult and complex instruments to create – focusing on learner preferences and attitudes ◦ WHAT – Direct learner statement, observation, or similar inference Psychomotor Domain ◦ HOW – Typically require the learner to demonstrate steps to show understanding of a universal concept ◦ WHAT – Rubric, checklist, rating scale, or direct demonstrations
- 6. Determining Mastery Levels In order for learners to “master”, they must achieve a certain level of performance. Mastery is equivalent to the level of performance normally expected from the best learners. Statistical Sufficient opportunities should be provided to perform the skill so that it is nearly impossible for correct performance to be result of chance alone. As a general principle, mastery level for any performance should be considered with respect to both evaluating the performance at the point in time and enhancing the learning of subsequent, related skills in the course. The best definition of mastery is the level requires in order to be successful on the job.
- 7. Writing Assessment Item You should write an assessment item for each objective whose accomplishment you want to measure. Steps to follow when writing a criterion assessment item: ◦ Read the objective and determine what it wants someone to be able to do (i.e., identify the performance). ◦ Draft a test item that asks students to exhibit that performance. ◦ Read the objective again and note the conditions under which the performing should occur (i.e., tools and equipment provided, people present, key environmental conditions). ◦ Write those conditions into your item. ◦ For conditions you cannot provide, describe approximations that are as close to the objective as you can imagine. ◦ If you feel you must have more than one item to test an objective, it should be because (a) the range of possible conditions is so great that one performance won’t tell you that the student can perform under the entire range of conditions, or (b) the performance could be correct by chance. Be sure that each item calls for the performance stated in the objective, under the conditions called for. If you follow these steps and still find yourself having trouble drafting an assessment item, it is almost always be because the objective isn’t clear enough to provide the necessary guidance.
- 8. Types of Test Item Criterion Goal- Centered Focusing on the objectives of the instructional Learner- Centered Focusing on the differentiated needs of learners Context- Centered Focusing on the environment in which learning occurs ass well as where direct application (i.e. performance) ultimately occurs Assessment- Centered Focusing on all aspects of the assessment design and creation
- 9. Setting Mastery Criteria What is the proper number of items needed to determine mastery of an objective? How many items must leaner answer correctly to be judged successful on a particular objective?
- 10. Types of Items You should select the type of item that gives learners the best opportunity to demonstrate the performance specified in your objective. Possible test items include: • Essay • Fill-in-the-blank • Completion • Multiple-choice • Matching • Product checklist • Live performance checklist
- 11. Test Item Format To select the best type of item from among those that are adequate, consider such factors as the response time required by learner, the scoring time required to analyze and judge the answers, the testing environment, and the probability of guessing the correct answer.
- 12. Writing Directions The title suggests the content to be covered. A brief statement explains the objectives or performance to be demonstrated and the amount of credit that will be given for a practically correct answer. Learners are told whether they should guess if they are unsure of the answer. Instructions specify whether words must be spelled correctly to receive full credit. Learners are told whether they should use their names or simply identify themselves as members of a group. Time limits, words limit, or space limits are spelled out. Directions for performance and products should clearly describe what is expected of the learners. You should include any special conditions and decide on the amount of guidance you will provide during the assessment.
- 13. Evaluating Tests and Test Items The designer should ensure the following: ◦ Test directions ◦ Each test item ◦ Conditions ◦ The response methods ◦ Appropriate space, time, and equipment After writing the test, the designer should administer it to a student or individual who will read and explain aloud what is meant by both the directions and questions and respond to each question in the intended response format. The designer should keep in mind that tests measure the adequacy of (1) the test itself (2) the response form (3) the instructional materials (4) the instructional environment and situation (5) The achievement of learners
- 14. Developing the Instrument When assessing performances, products, or attitudes you will need to create an assessment instrument to help you evaluate the performance, product, or attitude. Dick and Carey offer five steps to creating this instrument: Identify the elements to be evaluated Paraphrase each element Sequence the elements on the instrument Select the type of judgment to be made by the evaluator Determine how the instrument will be scored
- 15. Assessment of Performances, Products, and Attitudes Writing directions Developing the instrument Identify, paraphrase, and sequence elements Developing the response format: Checklist Rating Scale Frequency Count Scoring procedure
- 16. Using Portfolio Assessments Portfolios are collections of work that together represent learners’ achievements over an extended period of time. This could include tests, products, performances, essays, or anything else related to the goals of the portfolio. They allow you to assess learners’ work as well as their growth during the process. As with all other forms of assessment, whatever is included in the portfolio must be related to specific goals and objectives. The choice of what to include can be decide on entirely by the teacher, or in cooperation with students. Assessment of each portfolio component is done as it is completed, and the overall assessment of the portfolio is carried out at the end of the process using rubrics. In addition, learners are given the opportunity to assess their own work by reflecting on the strengths and weaknesses of various components. Portfolios can also be used as part of the evaluation process to determine what students did and did not learn, and then that information can be used to strengthen the instruction.
- 17. Evaluating Congruence in the Design Process One of the most crucial aspects of the assessment phase of the design process is to be able to evaluate the congruence of the assessment against the objectives and analyses that have been performed. Remember that this is a systematic approach to instructional design, which means that every step in the process influences subsequent steps. As such, all of your skills, objectives, and assessment items should be parallel. One way to clearly represent this relationship is to create a three-column table that lists each of the skills from your instructional analysis, the accompanying objective, and the resulting assessment item. At the bottom of the table you would finish up with your main instructional goal, the terminal objective, and the test item for the terminal objective. Design Evaluation Chart Skill Objective Assessment Item (s) 1 Objective 1 Test Item 2 Objective 2 Test Item 3 Objective 3 Test Item Instructional Goal Terminal Objective Test Item
- 18. Reference Dick, W., Carey, L., & Carey, J. O. (2009). The systematic design of instruction. (7th ed., pp. 130-163). Upper Saddle River, NJ: Pearson.
- 19. Summary A criterion-referenced assessment is composed of items or performance tasks that directly measure skills described in one or more behavioral objectives. Leaner-centered assessments are to be criterion-referenced. This type of testing of test is important for evaluating both learners; a process and instructional quality. Assessment Self-Reflection Think about how you have used assessments in the past to either gauge your personal knowledge and skill levels or to analyze the aptitudes of your learners. Do you feel the assessments helped you to gain a better understanding of your knowledge and that of your learners? If so, why, and if not, why not?
- 20. THE END CAROLYN JENKINS-HAIGLER