Evaluating Web-Based Learning Tools
- 1. Evaluating the Effectiveness of Web-Based Learning Tools for Online Learning Dr. Robin Kay UOIT Oshawa, Canada Website: faculty.uoit.ca/kay/home
- 2. UOIT • Started in 2003 • 6000+ Students • Laptop based university • Masters and PhD programs online
- 3. Overview • What is a Web-Based Learning Tool? • Why do we need to Evaluate WBLTs? • Previous Research • WBLT Evaluation Scale • Using the Scale • Study • Results • Conclusions
- 4. What is a Web-Based Learning Tool? • Sample 1 – Tool - Boyle’s Law • Sample 2 – Practice - Animal Cells • Sample 3 – Demonstration – Trigonometry Which one was the best tool?
- 5. Brief Definition of WBLT “Interactive tool that supports the learning of specific concepts by enhancing, amplifying, and/or guiding the cognitive processes of learners”
- 6. Why Evaluate WBLTs? 1. Quality assurance 2. Reduces search time 3. Encourages effective reuse
- 7. Previous Evaluation Research Typical Pattern • Informal & anecdotal • Limited number of WBLTs evaluated • Reliability and validity missing Evaluation Studies • Reviewed research and key models • List of Criteria
- 8. WBLT Evaluate Scale Construct Criteria Interactivity • Meaningful Interactions • Control • Multimedia that adds to learning Design • Consistency • Layout • Labeling • Readability Engagement • Quality of feedback • Attractive • Graphics (not video) • Learning modes • Motivation Usability • Natural to use • Orientation • Navigation cues • Instructions • Language level
- 9. Using the Scale • Sample 1 – Tool - Boyle’s Law • Use the Hand out to evaluate: ▫ Interactivity ▫ Design ▫ Engagement ▫ Usability
- 10. LOEM Study 4 Raters • 4 teachers – 3 days of training using scale • Inter-rater reliability – 94% to 96% Students • 1113 students (588 males, 525 females) • Secondary School • 64 different classrooms • 44 different WBLTs
- 11. Results - Reliability Construct Number of Items Reliability Interactivity 3 0.70 Design 4 0.74 Engagement 5 0.77 Usability 5 0.80
- 13. Results – Student Evaluations Construct S-Learn S-Quality S-Engagement Interactivity 0.06 0.08 -0.08 Design 0.25 * 0.30 * 0.20 * Engagement 0.24 * 0.32 * 0.27 * Usability 0.28 * 0.27 * 0.15 **
- 14. Results – Teacher Evaluations Construct T-Learn T-Quality T-Engagement Interactivity 0.06 0.19 * -0.10 Design 0.15** 0.19 * 0.06 Engagement 0.33 * 0.20 * 0.29 * Usability 0.12 0.40 * -0.13
- 15. Results – Student Performance Construct Student Performance Interactivity 0.28 * Design 0.43 * Engagement 0.37 * Usability 0.40 *
- 16. Conclusions • WBLT Evaluation Scale ▫ Fair reliability ▫ Good validity ▫ Agreed with student and teacher perceptions ▫ Predicted student performance
- 17. Practical Implications • When evaluating WBLTs ▫ Interactivity ▫ Design ▫ Engagement ▫ Usability • Can be used to evaluate WBLTs ahead of time to save educators time • Starts a more rigorous discussion about quality of WBLTs
- 18. Future Research - Resources Future Research ▫ Instructional Design ▫ Clark & Mayer (2008) Website ▫ faculty.uoit.ca/kay/wblt Contact Information ▫ robin.kay@uoit.ca