Using Learning Analytics Data to Understand Project-Based Learning
1 like394 views
The document provides background information on the Globaloria program, which aims to develop students' contemporary learning abilities through game design. It discusses implementation of Globaloria in several US states, including West Virginia. Specifically, it examines data from a high school class that participated in Globaloria during the 2010-2011 school year, focusing on one team called the Carrot Wizards. The data sources provided allow analysis of factors at different levels (e.g. individual, team, classroom) that may have influenced student learning outcomes. Preliminary analysis of wiki log files shows patterns of wiki use over time varied across teams, with the Carrot Wizards team exhibiting more early and late activity than others.
Using Learning Analytics Data to Understand Project-Based Learning
- 1. Using Learning Analytics Data to Understand Project-Based Learning The Case of the Carrot Wizards in Globaloria- West Virginia ICLS 2012 Pre-Conference Data Workshop *** Rebecca Reynolds, Ph.D. Assistant Professor, Library & Information Science Rutgers University School of Communication and Information
- 2. Background Info • The following slides provide background on the Constructionist / design-based research innovation, “Globaloria,” being implemented by a NYC non-profit called the World Wide Wokrshop, in several US states with grant and state education department support (WV, TX, CA, FL, NY)
- 3. Globaloria Game Design Program Learning Objectives: Cultivate the Six Contemporary Learning Abilities (6CLAs) Developing games in a social learning system offers students Constructionist, participatory design and programming experiences leading to dispositions, practices and expertise necessary for successful engagement in today’s digital, knowledge-based economy
- 4. The Six Contemporary Learning Abilities (6CLAs): Developing games in a social learning system offers participatory experiences leading to practices necessary for engagement in today’s knowledge-based economy
- 5. Globaloria is currently active in 4 U.S. states: CA, TX, FL, WV
- 6. Globaloria-California California, 2011/2012: • Supported by $1M Knight Foundation grant • “Formal” implementation at 2 middle schools (67 students) • “Informal” implementation at 2 Boys and Girls Club community centers (38 students) • First year of California participation • Potential to compare implementation context variable differences (time on task; provision of grades vs. none; linkage to core curriculum vs. open choice game topics, etc.)
- 7. Globaloria-TX East Austin College Prep, 2011/2012: • 1 charter middle school • 3 grades (6th, 7th, 8th) • ~250 students • 3rd year of school’s participation • 2 educators • Unique longitudinal implementation (all EACPA students take Globaloria daily, for credit and a grade for 3 years)
- 8. Globaloria-West Virginia2011/2012: • 5th year of Globaloria implementation in West Virginia • Individual schools and teachers range in N of participation years • Majority of students, 1 year only • 17 middle schools • 50 high schools • All in-school implementation, daily, for credit, for grade • Mix of elective game design classes, and required classes linked to the core curriculum • Multi-level analysis and implementation context variable differences increasingly being measured (e.g., teacher N of participation years; extent of teacher training; student game subject focus;
- 9. Domains of Learning and Expertise • Constructionist digital literacy (skills needed in knowledge economy => 6-CLAs) • Computational thinking through game design in Flash and programming in Actionscript • Core curricular subject matter: – When game subjects are linked to core curriculum and students deepen knowledge about topic through online research and design • STEM career interests: Technology & Engineering; Computer Science • Motivation, Affect, Attitudes, Life Choices, New Possibilities and Horizons, esp. for disadvantaged students
- 10. Flash software, Wiki Environment, Curriculum, Tutorials “Hands On” Training Sessions (virtual, local) • Globaloria Academy – In-person, intensive trainings (3) • Online Mini Webinars - Web-based workshops (7) Globaloria Mentors Program Experienced educators take on a leadership role by supporting other educators “24/7” Virtual Support • Expert Support via wikis, blogs, email, WebEx • Educator Community Development – private educators community wiki, peer-to-peer mentoring, weekly educators newsletter, sharing teaching & learning reports Rewards and Recognition •Teachers: Stipends and Graduate credits are earned •Students: Nationally-Recognized Game Design Competitions Learning Supports for Students and Educators:
- 11. Syllabus integrates learning about the game subject topic and game programming and mechanics
- 12. Syllabus integrates learning about the game subject topic and game programming and mechanics Syllabus Topics Addressing Game Subject Syllabus Topics Addressing Game Programming and Mechanics www.myglife.org/usa/wv
- 13. Syllabus integrates learning about the game subject topic and game programming and mechanics
- 14. Data Collection for Research Occurs via 6 Main Technology and Curriculum Platforms Developed by the World Wide Workshop
- 15. Data Collection for Research Occurs via 6 Main Technology and Curriculum Platforms Developed by the World Wide Workshop
- 16. Data Collection for Research Occurs via 6 Main Technology and Curriculum Platforms Developed by the World Wide Workshop
- 17. Data Collection for Research Occurs via 6 Main Technology and Curriculum Platforms Developed by the World Wide Workshop
- 18. The Globaloria Learning Formula: Discovery-based project-based, student- centered, social learning
- 19. Some Globaloria project research goals at a high level • Understanding the influences and forces at play in this social learning system; mapping the terrain from a social science perspective • Design-based research: Iterative improvement of the social learning system; generalization towards learning theory, and the instructional design of effective learning innovations Connecting multiple levels of analysis in our research will be necessary to accomplish the above . . . And ideally, the below: • Evaluative Diagnostics: – Use predictive modeling to identify trajectories of student success / failure across time (what factors at multiple levels of analysis influence student learning?) – Apply predictive findings towards development of diagnostic tools for early intervention that teachers can use for formative assessment • Summative Evaluation: Developing tools for program evaluation and student assessment
- 20. 2 Main Strands of Research Quantitative multi-level analysis modeling (w. Ming Ming Chiu) • Purpose: Model the complex relationships among variables of importance. Currently, DV is evaluation scores for games based on content analysis of their quality • Present data sources: – Teacher and student pre / post surveys – Teacher progress reports – Content analysis evaluation of team games – Wiki log files and behavioral site metrics data (edits, uploads) – Other data TBA (e.g., game subjects; teacher grades; student role-taking; resource use) Qualitative analysis, comparison of cases (w. Cindy Hmelo-Silver) • Purpose: investigate and explain dynamics / inter-relationships among variables; identify key phenomena driving the system; add to quantitative model as relevant • Present data sources: – Semi-structured teacher and student team interviews at two timepoints this past semester (18 middle school teams, TX & WV, and their educators) • Main themes: Teacher/peer/e-learning/non-profit scaffolding; student problem self-structuring; student/teacher resource use; areas of challenge/enjoyment; teamwork processes – Future: Participant observation in situ in the classroom
- 21. Challenging Assumptions • To what extent do complex multi-level phenomena and dynamics in this system break down into discrete variables that can be investigated for their inter- relationships in quantitative modeling? Is the phenomena in such a social learning system better described quantitatively? Qualitatively? Mixed methods? How can varying analyses inform each other? – Perhaps something we can discuss today
- 22. Analytical difficulty Statistical Discourse Analysis strategy Outcome variable Binary Infrequent Teams within classrooms (Nested data) Multiple outcomes Independent variables Indirect, mediation effects False positives Logit (Kennedy, 2008) Logit bias estimator (King & Zeng, 2001) Multilevel analysis (Goldstein, 1995) Multi-outcome analysis (Goldstein, 1995) Multilevel M-Tests (Krull & MacKinnon, 2001) Two-stage linear step-up procedure (Benjamini et al., 2006)
- 23. Mean pre-intrinsic motivation +0.176 * Mean increase in intrinsic motivation +0.163 * Mean SWF flash uploads per month +0.166 * Mean age –0.276 *** Team Structure Design mixed genre game +0.833 ** Team Processes Design Game play Concept Mean self- learning hours Programming –0.452 *** Teacher Outcome
- 24. Analysis highlights 3 levels Classroom (teacher) Team (mean of group members) Team process (mean of team processes) Test hypotheses about outcomes w/ explanatory variables at lower levels --characteristics of samples Mean, SD, max, min, … Model multiple outcomes simultaneously
- 25. Qualitative analysis 1. What are the theoretical instructional design assumptions implicit in the design and development of Globaloria? – Guided discovery-based learning, building on Constructionism – Scaffolding provided via a range of structures (explored in research) 2. How are students at different schools interacting with the range of scaffolds provided? Autonomous/controlled/task-driven – A focus on student self-structuring of problems and resource uses to support their learning, at various stages of the process 3. Linking Scaffolds, Problem-Solving, and Resource Use: – To what extent does it appear that the structuring elements / scaffolds above help students meet learning goals: conceptualize the idea for a game, develop a prototype, implement a game in Flash, learn programming in Actionscript, complete the final game, do this in a team context (combining files), and upload the final game to the wiki? 4. Does this process appear to bring about other outcomes besides discrete CLAs/Constructionist digital literacy, and/or other domains of expertise (computational thinking; STEM career interest) – Resourcefulness?
- 26. Approach to the ICLS 2012 Data Workshop What can we learn about multi-level phenomena from an in-depth focus on 1 class of 2010/2011 game designers, using an eclectic array of data sources? • Investigate the experience of a single case study team of game designers from TVHS, consisting of 3 HS students who won the annual game design competition in the 2010/2011 school year with their Civics content game, “Henry the Hedgehog” – Understanding team processes can expose some of the values/norms built into the system (what makes a good game?) • Investigate relationships among variables that appear influential to learning. • Consider this case in context, given that similar data are available for over 2000 students across 3+ years • Apply findings to ongoing research being conducted
- 27. Data Sources Provided to Participants, and Corresponding Levels of Analysis Level of Analysis Local context Data source Filenames Innovation Curriculum Description of Globaloria syllabus,program overview 2.Curriculum.pdf ICLS2012_dataworkshop.pptx E-learning platform Description of features of e-learning platform; screenshots; wiki log files (trace data of student activity on the wiki) 2.Curriculum.pdf ICLS2012_dataworkshop.pptx 13. Aggregate wiki log _ contentanalysis.xlsx System WV Department of Education Game competition press release; information on judges and judging criteria. 14.game_comps_press_2011-1.pdf SampleWikiCurricLesson_Drag and Drop.docx WikiScreenshots folder Organizational School Description of the school 3.Case Study Data File, Carrot Wizards.docx ICLS2012_dataworkshop.pptx World Wide Workshop non-profit Description of non-profit, goals 3.Case Study Data File, Carrot Wizards.docx ICLS2012_dataworkshop.pptx Class Teacher Progress report from teacher offering reflection on PD experiences, learning and teaching; and observations of students -- individuals and teams 4.teacherprogress report.doc Team Game design teams Team key (who is in what team); Carrot Wizards case study report; evaluation of team games; wiki log files 5.teammembers_fullclass_tvhs_y4.csv 7.1.CarrotWizardsFINALGAME.docx 8.yr4_globaloria_eval_researchfile_TVHS.xlsx 13. Aggregate wiki log _ contentanalysis.xlsx Individual Student participants Pre- and Post-Surveys; student blog URLs; case study report; wiki log files 9.yr4_globaloria_postsurvey_qualitative_TVHS.xls 10.yr4_globaloria_presurvey_qualitative_TVHS.xls 11.blogURLsforCarrotWizards.docx 13. Aggregate wiki log _ contentanalysis.xlsx Artifact Student games; student wiki work Actual team games; wiki screenshots; game evaluation 6.File_Cwfinal2atyler.fla 7.1.CarrotWizardsFINALGAME.docx 7.File_Cwfinal2atyler.swf 3.Case Study Data File, Carrot Wizards.docx WikiScreenshots folder 8.yr4_globaloria_eval_researchfile_TVHS.xlsx
- 28. Further context about TVHS • WV high school class at TVHS took Globaloria as a game design elective, meeting daily for 90 minutes per day, for credit and a grade across an entire school year (2010/2011) • This was TVHS and educator A. Leef’s first year in the program implementing Globaloria; she participated in the standard Globaloria Academy PD training and ongoing offerings throughout the year. • Students in 8 teams of 1, 2 or 3, created a total of 8 games about a range of topics relating to either Civics or STEM • 2 games focused on Civics: Carrot Wizards’ Henry the Hedgehog game, and Skull Crusher’s game Blast. • Henry the Hedgehog game won the Globaloria Annual Civics Game Design competition in the 2010/2011 school year
- 29. Analysis: Narrowing in on Team Process step of quantitative multi-level analysis, using Wiki log files • Behavioral trace log file data available, measuring individual-level edits to wiki pages, and uploads of game artifacts, graphics, .FLAs and .SWFs. • Research Questions: – How do edits / uploads of teams at TVHS vary across time? • Totals, team means, indivs-within-teams – General observations about wiki use patterns – What assumptions/limitations associated with using frequency data in multi-level analysis? – What stories do the data tell, that the aggregate frequency numbers miss?
- 30. Preliminary Researcher Analysis of Data Sources: Wiki log files • Edits to team pages (main locus on the wiki for team-level activity): • Game project file uploads
- 31. Team Rankings, Total Sum Evaluation Score Team Name:TOTAL SUM Carrot_Wizards 57 Shock-a-holics 53 LEET_HAX 51 Bull_Loany_Productions 50 Gummy worms.(: 49 Thomas the Train's Pancakes of Pain 49 TEAM_SKULL_CRUSHER 46 TOMORROW 19
- 32. Aggregate Sum Results, Edits Team Name N of Team Members Total Edits Mean Edits Carrot_Wizards 3 161.00 53.67 Gummy_worms.(: 2 144.00 72.00 LEET_HAX 1 66 66 Shock-a-holics 3 91 30.33 TEAM_SKULL_CRU SHER 2 60 30 Thomas_the_Train 's_Pancakes_of_Pa in 3 60 20 TOMORROW 1 12 12
- 33. Descriptive Results Total Edits to Wiki Team Page Week of 9/20/10 – week of 5/9/11 0.00 20.00 40.00 60.00 80.00 100.00 120.00 140.00 160.00 180.00 Total edits Mean edits
- 34. Total N of Edits to Team Wiki Pages, Week of 9/20/10 – week of 5/9/11 How does team activity vary across time? • Carrot Wizards appears to have invested greater effort in editing their team wiki page at the beginning than other teams. Evidence of possible impact of early planning. • Carrot Wizards and Gummy Worms appear most consistent in their wiki editing across time • Carrot Wizards appears to have exerted more effort towards the end of the school year than other teams Note: 1 outlier data point from a single day by a single student in 2010w12, Leethax team omitted
- 35. Mean Edits to Team Wiki Pages, Week of 9/20/10 – week of 5/9/11 (Mean = Total N of page edits / N of team members) How does standardizing for N of team members influence patterns? • Gummy worms had fewer team members, thus logged more average activity with consistency as a team Note: 1 outlier data point froma single day by a single student in 2010w12, Leethax team omitted
- 36. Explaining Gummy Worms greater N of edits (individuals within teams) • Consistency in editing among both team members
- 37. Explaining Gummy Worms greater N of edits (individuals within teams) • Tyler was primary wiki team page editor. The other 2 team members only did so in S2 at the end. • What is the difference in team dynamic, between having 2 vs. 3 team members?
- 38. Total N of File Uploads, Week of 9/20/10 – week of 5/9/11 How does team activity vary across time? • All teams have greater frequency/consistency of uploads in S1; uploads become more concentrated during S2 • File types vary to a greater extent in S1 (includes more static graphics files) • Carrot Wizards and Gummy Worms files are mainly SWFs and FLAs in S2
- 39. Explaining Gummy Worms greater N of uploads (individuals within teams) • Brianna uploads most frequently and consistently • Nicole outlier in week 30
- 40. Carrot Wizards individuals-within-team Uploads • Relative consistency among team members • What was occurring at week 29-34 in the program curriculum to spark high uploads among these 2 teams?
- 41. Connecting team and individual levels of analysis • How is wiki work distributed across individual team members? – Procedure: • For Carrot Wizards only, reviewed log files across time, by team member • Review actual wiki content edits made by team members across time • Note categorical types of wiki content edits • Make observations regarding individual activity within the team – Link inferences to team-level findings above
- 42. Carrot Wizards distribution of individual contributions to wiki
- 43. Carrot Wizards distribution of individual contributions to wiki
- 44. Carrot Wizards distribution of individual contributions to wiki
- 45. Carrot Wizards distribution of individual contributions to wiki
- 46. Carrot Wizards distribution of individual contributions to wiki
- 47. Connecting Wiki activity to Evaluation Scores • To what extent do patterns of wiki activity relate to evaluation outcomes? – This is a next step in our analysis. – We welcome the input of other researchers
- 48. Next Steps • Summarize findings on individual contributions within the team context • Consider results in relation to game evaluation findings
- 49. Opportunities for Workshop Participants • Conduct social network analysis / sequential interaction analysis on individual and team level wiki activity results for TVHS teams – Raw aggregated wiki log file and evaluation outcome data are available in the Excel file, “13. Aggregate wiki log _ contentanalysis.xls” • Relate wiki raw frequency and social network analysis findings to final game evaluation outcomes • Explore relationships between qualitative findings in Dropbox, and results reported herein
Editor's Notes
- #11: How do we plan to grow exponentially in a way that is sustainable? Strong and innovative Professional Development programs for educators, principals and students ensure the community can self-manage, grow and develop.Hands-on training: Mentor program – cascading and taken to scaleVirtual support – sustainable, scalable – walking our talkRewards & Recognition – we pay
- #12: In addition to the amazing impact we have seen over the past few years researching and evaluating the Globaloria program we…We are in unique position of having everything we do be rooted in years of academic research and real-world practice. In the 1980s and 1990s, while at Harvard and the MIT Media Lab, Idit really founded the concept that children learn best by designing – when they are programming computers instead of computers programming them. Ground breaking research with children in Bronx, showing how software designing could change their relationship to education and their engage them in learning in a way that had not been seen before
- #13: In addition to the amazing impact we have seen over the past few years researching and evaluating the Globaloria program we…We are in unique position of having everything we do be rooted in years of academic research and real-world practice. In the 1980s and 1990s, while at Harvard and the MIT Media Lab, Idit really founded the concept that children learn best by designing – when they are programming computers instead of computers programming them. Ground breaking research with children in Bronx, showing how software designing could change their relationship to education and their engage them in learning in a way that had not been seen before
- #14: In addition to the amazing impact we have seen over the past few years researching and evaluating the Globaloria program we…We are in unique position of having everything we do be rooted in years of academic research and real-world practice. In the 1980s and 1990s, while at Harvard and the MIT Media Lab, Idit really founded the concept that children learn best by designing – when they are programming computers instead of computers programming them. Ground breaking research with children in Bronx, showing how software designing could change their relationship to education and their engage them in learning in a way that had not been seen before
- #19: We have taken all this learning and experience and established the cornerstone of the Globaloria Program – It’s Learning Formula. I want to take a minute on this slide: Self-Led Learning- students determine the needs of their learning as they progress through the process of designing their own games. It is purposeful, self-directed and hands-on learingPeer-to-peer learning – it is collaborative learning. Students take on the role of both educators and students. They have a critical role in their own learning and in the advancement of their learning community. They are invested. Co-Learning – we are breaking down the traditional hierarchy in the classroom and having students and educators learn together. Students don’t learn for the grade, but as part of a community of creators. Expert-guided learning – we do not divorce learning from the real-world…it is the real world. Students engage with real professional and see the relevance of their learning in today’s world.