ECGBL2015 Slides
Download as PPTX, PDF0 likes112 views
This document describes Dr. Jacob Habgood's research into evaluating game-based learning using a case study of his game Zombie Division. It discusses four designs of his classroom research studying the game, comparing intrinsic and extrinsic approaches. Design 4 found that an intrinsic group using teacher-led reflection had larger learning gains and gains that persisted longer according to delayed post-testing. The document also discusses the practical challenges of conducting educational game research in real classrooms.
ECGBL2015 Slides
- 1. Zombie Division: AMethodological Case Studyfor theEvaluation of Game-Based Learning Dr. Jacob Habgood Sheffield Hallam University, UK
- 2. Who am I? • Ex-game developer • Author • Teacher of PlayStation development • Researcher of zombies (or am I?) Jake
- 4. Zombie Division • PhD Research 2005/6 • Zombie count = 0 • Intrinsic Integration • Kafai (2001) • Results published in 2011 • Empirical evidence supporting an intrinsic approach as more educationally effective • Download the game from ResearchGate (not Win8+) Habgood & Ainsworth. The Journal of the Learning Sciences 20.2 (2011): 169-206 Habgood (2007) “The Effective Integration of Digital Games and Learning Content” “Chocolate-covered broccoli” Bruckman (1999) BAD Download and play it for yourself!
- 5. Research Design Asking the right research question…
- 6. Epistemology • GBL = “Broad Church” • Began in late 60’s early 70’s • Are we able to clearly demonstrate that GBL is effective yet? • Lack of high-quality empirical evaluations • “Effect on learning qualified as moderately positive” (Wouters et al., 2015)
- 7. HAA-PEE-RR ZOM-BEE HAA-PEE ZOM-BEE Educationally Effective? • Learning gains • Pre-post test • Motivation • Self-reported measures • Time-on-task • Meaningful comparisons • Improvement and/or engagement isn’t necessarily anything to shout about! + Before After
- 8. Making the Right Comparison • Classroom comparisons of learning • More effective than a cup of coffee? • Circular problem • Home comparisons of motivation • More time than what? • Circular problem Is GBL more effective than traditional classroom teaching? Is GBL as motivating as mainstream digital entertainment? Is classroom teaching more effective combined with GBL? Is GBL motivating enough to compete for time in the home?
- 9. Making the Right Comparison • “Status Quo”: Progress • Chocolate-Covered Broccoli • Loosely coupled learning and gameplay • “Intrinsically Integrated” • Tightly coupled learning and gameplay
- 10. Classroom Research The practicalities of “real world research”
- 11. Ethics • Duration of Intervention • Competing demands • Control group • 20 minutes per day for week • Age ratings • PEGI: Europe • ESRB: America • Recommend Google Play’s automated system (requires a developer account) ? Treatment group Treatment group Control group
- 12. Treatment Groups • Group size n >= 20 • 3 groups = 60+ participants • 2 classes per year group • <= 30 students per class • Combine year groups • Eldest from year below • Youngest from year above
- 13. Group Assignment • Randomised assignment • Small group sizes can lead to confounding factors • Matched Designs • Sorted by matched variable • Balanced dependant variable • Repeat by gender • Absenteeism • Delay between pre-test and intervention • Inevitable drop-out rate? 80% 79% 76% 71% 63% 58% 55% 52% 66% 67.5%
- 14. Gaming “Episode” • 120 minutes of play! • Familiar genre/controls • Zelda / Spyro • Focussed learning content • NOT a commercial product! • The inverse relationship between multiplication and division (National Curriculum)
- 15. Zombie Division The research design “journey”
- 16. Design 1: Learning Gains • Sixty-four 7-9 year olds • School day in ICT suite • Two conditions and control • Control = just chocolate! • Pre to Post-test design • Paper-based tests • 20 minutes playing per day x64 Extrinsic group Intrinsic group Control group
- 17. x64 Design 1: Learning Gains • No main effect of group! • Absenteeism didn’t help • Ceiling effect of pre/post tests • Making link between hundred square and division problems was too hard • Swapped for multiplication grid EEE-SEE!
- 18. Design 2: Time-on-task vs. alternative • Forty-four 7-8 year olds • School holidays in learning centre • Two conditions (no control) • Free choice • Zombie Division • BBC Website games • No absentees / ceiling effects • A single day (135 minutes) • Computerised test (fixed time, unlimited questions) • Automated group assignment x44 Extrinsic groupIntrinsic group
- 19. Design 2: Learning Gains • No main effect of group! • No absenteeism • No ceiling effects • Contamination • Competition between groups • Frustration with extrinsic ! x44 Extrinsic groupIntrinsic group MEEE WIN NO MEEE
- 20. Design 3: Time-on-task with free switching • Sixteen, 7-8 year olds • After school club (3x45 mins) • Free switching • Played both versions • Randomised order • No loss of progress • Huge preference for intrinsic • 61% of time vs. 8% • Deep design understanding • “subliminal learning with maths” x16
- 21. Design 4: Learning gains with teacher-led reflection • Sixty-four 7-8 year olds • School day in ICT suite • Two conditions and control • Control = just chocolate! • Pre, post, delayed-test design • Computer-based tests • 20 minutes playing per day • Additional reflection activities • Delayed test • Transfer tests x58 Extrinsic group Intrinsic group Control group Day 1 Computer-based pre-tests. Matched assignment: intrinsic, extrinsic & control. 2-3 20 minutes playing the game. 4 30 mins reflection + 20 mins play. 5-6 20 minutes playing the game. 7 Post-test 2 weeks 8 1 x 35 minutes playing the game. Delayed-test and transfer-test.
- 22. Design 4: Learning gains with teacher-led reflection • Intrinsic group differences! Larger overall percentage gains (15%pts).Percentage gains between post and delayed were significantly higher • Results published in 2011 • Habgood M. P. J. & Ainsworth, S. Motivating Children to Learn Effectively: Exploring the Value of Intrinsic Integration in Educational Games The Journal of the Learning Sciences 20.2 (2011): 169-206 Habgood & Ainsworth. The Journal of the Learning Sciences 20.2 (2011): 169-206 10 20 30 40 50 60 70 Pre Post Delayed PercentageScore Intrinsic Extrinsic Control
- 23. Process Data The practicalities of “real world research”
- 24. Process Data • Time-stamped log files • Post-hoc analysis • Baker, Ryan SJD, et al. "Modelling the acquisition of fluent skill in educational action games." User Modelling 2007. Springer Berlin Heidelberg, 2007. 17-26. • Behaviour analytics • E.g. in and out of rooms • E.g. pausing strategy
- 25. Input Streaming • Deterministic • Controller input streams • Replay against game exe • Pseudo-random numbers • 130 hours = 8MB storage! • Out of sync errors • Butterfly effect • Self-correcting data • Periodic position/orientation • Can’t help overall game state