Designing and Conducting Formative Evaluations
- 3. Formative Evaluation This is a form of evaluation that is used to improve program design by testing before or during a projects implementation. This evaluation ensures that the desired needs are being met, relevant feedback is being collected, and helps with the finalization of the chosen design.
- 4. Proactive Clarificative Interactive Monitoring When Pre-project Project development Project implementation Project implementation Why To understand or clarify the need for the project To make clear the theory of change that the project is based on To improve the project’s design (continual improvement) as it is rolled out To ensure that the project activities are being delivered efficiently and effectively Examples Literature Review Stakeholder Analysis Problem / Solution tree analysis Logframe Matrix Program Logic Semi-structured Interview Focus Group / ORID Project Diary Budget tracking Time tracking Questionnaire Dartboard Observation Categories of formative evaluationhttp://evaluationtoolbox.net.au/index.php?option=com_content&view=article&id=24&Itemid=125
- 5. Role of Subject-Matter, Learning, and Learner Specialists in Formative Evaluation In order to conduct a Design Review, you must have: Subject-Matter or the information or ideas that are discussed or dealt with in a book, movie, etc.: what something is about
- 6. Role of Subject-Matter, Learning, and Learner Specialists in Formative Evaluation The purpose of the review or evaluation is to ensure students are Learning: the activity or process of gaining knowledge or skill by studying, practicing, being taught, or experiencing something: the activity of someone who learns http://www.merriam-webster.com/
- 7. Role of Subject-Matter, Learning, and Learner Specialists in Formative Evaluation The content and activities that take place during the evaluation are reviewed by a specialist, a person who has special knowledge and skill relating to a particular job, area of study that ensures that the design supports its objectives.
- 8. One on One Learning One-to-one evaluation can be used to evaluate the intrinsic qualities of the design (clarity, usability, sequencing, completeness) and the learning effects (performance, practice, time, transfer).
- 9. One on One Learning A one-to-one evaluation can give you feedback from a learner that will help you improve your design. In on-to- one evaluation, you sit with learner as learner reviews instruction, observing how the learner uses it, noting learner's comments, probing about comments, and questioning the learner during/after instruction.
- 10. One on One Learning You should do one-to-one evaluation as early as possible, usually with roughs. Often you will do a sequence of 2-4 evaluations with the first focusing on obvious errors, later on intrinsic quality and later performance. One-to-one evaluations should be done before small group and field tests and before/during/after expert review.
- 11. One on One Learning Pros • you get the learners' point of view • the interactive evaluation gives you process data • it's easy, cheap, and quick • it's productive (Wagner, 1983; Robeck 1965; Lowe, Thurston, Brown 1983) • it can be used to rapidly test multiple versions of the design
- 12. One on One Learning Cons • the small number of users prevents you fro seeing patterns • it has low environmental validity because it's out of context, involves an strange 1-1 interaction; and usually involves a rough version of the design. • https://loft.io/guide/eeole/one-to-one-evaluation/
- 13. Small Group Evaluationhttps://loft.io/guide/eeole/small-group/ A small group evaluation can be used to quickly get you information about the learning impact and likability of your design. The whole point of your learning environment is to promote learning -- small group analysis is often the method in the design process that can provide the earliest evidence that the design is having the desired effect.
- 14. Small Group Evaluationhttps://loft.io/guide/eeole/small-group/ In small group evaluation, you test an early version of the learning environment with a representative sample of learners. Usually, the instruction is taught by a teacher while the design team records observations and gathers data. Small group evaluations typically include tests and questionnaires.
- 15. Small Group Evaluationhttps://loft.io/guide/eeole/small-group/ Advantages • they can be cheap • you can get lots of data quickly • they can require less interaction than one-to-one evaluation because the tester may not necessarily need to constantly question the participants
- 16. Small Group Evaluationhttps://loft.io/guide/eeole/small-group/ Disadvantages • may give less process/interaction data than one-to-one evaluation • may not catch a problem as quickly as in one-to-one evaluation in which you are constantly, directly monitoring a single learner • may not allow you to ask why something is not working • can be expensive if you must pay learners • may provide more data than you have time to analyze.
- 18. During the field trial, the materials are tested in a "real- world situation".
- 19. The purpose of the field trial is to ensure that all of the problems from earlier evaluations have been corrected and to see whether the instruction works in the intended context.
- 20. The instructional materials should be tested in a "real world" situation with a typical group of learners. The procedures followed should be those outlined in the instructional materials..
- 21. Finally, any final revisions can be made. http://eduscapes.com/instruction/13.htm
- 22. Formative Evaluation of Selected MaterialsDoes not always follow the traditional evaluation process. Usually results in the creation of supplemental materials such as study guides, workbooks, and syllabi.
- 23. Formative Evaluation of Selected MaterialsEvaluating selected materials includes Expert Judgment Studies. These studies help to determine if the current course of study is meeting requirements and if other courses or additional supplemental also meet requirements.
- 24. Formative Evaluation of Instructor-Led Instruction These evaluations are focused on the effectiveness of current instruction and how it can be improved.
- 25. Formative Evaluation of Instructor-Led Instruction Field trial instructors pay attention to these following learner knowledge: entry skills, prior topic knowledge, posttest scores, and learner attitude.
- 26. Formative Evaluation of Instructor-Led Instruction These field trials are helpful as they provide instructors with learner feedback and provide a practice medium for instructional procedures.
- 28. Context Concerns • Instructors should ensure equipment is running and stored properly for next instructor • Students should be encouraged to work within a quiet and education focused environment
- 29. Concerns about Learners Before instruction, administer entry level testing in order to gauge learner knowledge. Results should be focused on those who have mastered the skills
- 30. Concerns about Learners Those who have mastered the skills should continue as learners but the focus should be on obtaining a 100 percent performance ranking.
- 31. Concerns about Learners These learners can be selected as “subject matter sophisticates “who can help others obtain 100 percent performance.
- 32. Concerns about Learners Those who do not pass the entry level testing should be included in instruction in order to help identify what skills are required for learning and what skills need to be included as entry level skills.
- 33. Concerns about Formative Evaluation Outcomes Be prepared to find problems and concerns with your instructional design.
- 34. Concerns about Formative Evaluation Outcomes Understand that the positive feedback is limited and can only be related to a limited group of learners.
- 35. Concerns about Formative Evaluation Outcomes Be objective about the instruction design in order to spot areas that need improvement
- 36. Problem Solving during Instructional DesignWhen a problem arrives dealing with your Instructional Design, use learner information to determine the answer. Information derived from formative evaluation and other methods provides data that can be used to improve instruction.