Learning Analytics Design in Game-based Learning

@playfulMIT
Learning Analytics Design in
Game-based Learning
José A. Ruipérez Valiente - @JoseARuiperez
YJ Kim - @yjkimchee

@playfulMIT
Introductions
Who are you and why am I here?

@playfulMIT
playful.mit.edu
@playfulMIT

@playfulMIT
What we believe in

@playfulMIT
Workshop organizers
José A. Ruipérez-Valiente
- Postdoc @ MIT
- Engineer, focus on edtech
- 6 years working in learning
analytics across many objectives
and contexts
- Currently focused in large scale
trends in MOOCs and games for
learning
YJ (Yoon Jeon) Kim
- Executive Director Playful Journey
Lab located at MIT Open Learning
- Assessment scientist
- Focus on games and playful
approaches for assessment
- Just had a baby!

@playfulMIT
Overview of the Workshop
PART I:
- Overview of game-based learning and assessment
- Example with Physics Playground and evidence-centered design approach
- Activity: mapping game evidence to constructs
PART II:
- Introduction of learning analytics
- Example with Shadowspect and Learning Analytics approach
- Activity: mapping evidence to data and design algorithmic machinery
- Share out by each group

@playfulMIT
How does the workshop relate to TEL topics?
- Game-based Learning
- Game-based Assessment
- Learning Analytics
- … and Design (which is transverse to numerous areas and applications)

@playfulMIT
FIRST PART:
What makes good games that can afford analytics and how to avoid junk in junk out?

@playfulMIT
Assessment is a process of reasoning
from evidence. Therefore, an
assessment is a tool designed to
observe students’ behavior and
produce data that can be used to draw
reasonable inferences about what
students know.
~ Bob Mislevy

@playfulMIT
A game is a voluntary interactive
activity, in which one or more players
follow rules that constrain their
behavior, enacting an artificial conflict
that ends in a quantifiable outcome.
~Eric Zimmerman (2004)

@playfulMIT
Why Games?
● Games are “flexible enough for players to
inhabit and explore through meaningful
play” (Salen & Zimmerman 2004, p. 165)
(deep learning)
● Learners have more freedom related to
how much effort they choose to expend,
how often they fail and try again (Osterweil,
2014) (real life)

@playfulMIT
Why Games for Assessment?
● Games incorporate multiple pathways to solution(s) → Rapid/numerous data
streams → Rich and comprehensive student models
● Use complex and authentic problems → construct validity esp for hard-to-
measure constructs
● Games are motivating and engaging → accurate assessment (Sundre &
Wise, 2003)
● It doesn’t feel like assessment (i.e. stealth assessment)

@playfulMIT
Design, Development and Evaluation Process of Game-based
Assessment

@playfulMIT
Game Design vs. Assessment Design

@playfulMIT
Our simplified case scenario right now
Evidence Constructs
map

@playfulMIT
Example of Evidence-centered
design with the Physics Playground

@playfulMIT
Video Physics Playground
See video of Physics Playground here

@playfulMIT
Physics Playground: Competencies and Evidence

@playfulMIT
Physics Playground: Data example

@playfulMIT
Physics Playground: Expert Assessment of Constructs

@playfulMIT
Now it’s your turn!

@playfulMIT
Let’s form groups by game!
If you already have a ed game you would like to work on, let me know. Otherwise:
- Shadowspect
- Physics Playground (user = demo1, pw = demo1pass)
- Quandary
- Spanish Bingo
- Make a Number
- Argument Wars
- Want to try other game? Some ed game aggregators MathPlayground, iCivics or ABCYa

@playfulMIT
Activity!
- Play the game for about 15 minutes to understand the domain and mechanics
- Select and read about a construct that could be measured in the game:
- Content knowledge: e.g., if the game is about mathematics, multiplication, summation.. More
detailed approach is checking the Common Core Standards by age:
- Behavioral constructs: Off-task, persistence, productive struggle, non-constructive behaviors
- Cognitive skills: Spatial reasoning, creativity, memory, attention...
- Find what Evidence in the game is related to that construct
- What changes would you make in the game mechanics to facilitate measuring those
constructs?

@playfulMIT
Share out
- Recommended format:
a. What was your game about?
b. For each constructs that you have identified as measurable:
- Name the construct and describe it
- What is the evidence you find in the game related to the construct?
- Do you recommend a change in the game mechanics to improve the
evidence?
c. What did you learn with this activity?

@playfulMIT
BREAK
Next part will focus on designing how to engineer the analytics to inform your
constructs

@playfulMIT
SECOND PART:
How can we effectively use learning analytics for inference in games for learning?

@playfulMIT
Recap
What we did:
- Introduced why games and assessment go well together
- Presented an example of how to apply Evidence-centered Design in a game
- Did an activity to map the evidence in a game to constructs
What we are doing now:
- Introduce LA and why is a good approach for learning games
- Present example on classical data-driven LA in a game
- Activity to map evidence to data (schema), and constructs to features (algorithms)

@playfulMIT
The Broad view of Learning Analytics
…collection, analysis and reporting of data about learners and
their contexts, for purposes of understanding and optimising
learning and the environments in which it occurs…
Source: First Learning Analytics
and Knowledge Conference

@playfulMIT
The Learning Analytics data-driven Process

@playfulMIT
Our simplified case scenario now updates to:
Evidence Constructs
map
Data Features
data schema inform
algorithms

@playfulMIT
Example of Learning Analytics
approach with Shadowspect

@playfulMIT
Video of Shadowspect
See video of Shadowspect here

@playfulMIT
Efficiency construct
- Efficiency is the ability to do things well, successfully, and without waste. It
often specifically comprises the capability of a specific application of effort
to produce a specific outcome with a minimum amount or quantity of
waste, expense, or unnecessary effort (Wikipedia)

@playfulMIT
Evidence in Shadowspect related to efficiency
● Ability to do things well:
○ Solving puzzles correctly
● Expense or effort:
○ Time invested
○ Number of attempts to solve a problem

@playfulMIT
Mapping evidence into necessary data in Shadowspect
● We need: puzzles solved correctly, time invested and attempts
○ Necessary types of events for that:
■ puzzle_start (timestamp, student, puzzle_id)
■ leave_to_menu (timestamp, student, puzzle_id)
■ puzzle_attempt (timestamp, student, puzzle_id, correct)

@playfulMIT
How does data in Shadowspect actually looks like?

@playfulMIT
Algorithm to compute features from data (pseudo-code)
# note this is a VERY simplified version that do not aim to be the most effective implementation of this algorithm
computeEfficiencyFeatures(student):
student_events = getStudentEvents(student)
correct_puzzles_list = list(); number_attempts = 0; total_time = 0; puzzle_started_event = None
for event in student_events:
if(event[‘type’] == ‘puzzle_started’) then
puzzle_started_event = event
elif(event[‘type’] == ‘leave_to_menu’) then
total_time += (event[‘timestamp’] - puzzle_started_event[‘timestamp’])
puzzle_started_event = None
elif(event[‘type’] == ‘puzzle_attempt’):
number_attempts += 1
if(event[‘correct’] == True) then
correct_puzzles_list.add(event[‘puzzle_id’])
attempts_per_correct_puzzle = length(unique(correct_puzzles_list))/number_attempts
time_per_correct_ puzzle = length(unique(correct_puzzles_list))/total_time
return(attempts_per_correct_puzzle, time_per_correct_puzzle)

@playfulMIT
The previous general scenario
Evidence Constructs
map
Data Features
data schema inform
algorithms

@playfulMIT
Model for efficiency in Shadowspect
Evidence
● Correct puzzles
● Time
● Number attempts
Data
● puzzle_start
● leave_to_menu
● puzzle_attempt
data schema inform
computeEfficiency
Features(student)
Construct
Efficiency
Features
attempts_per_correct_problem
time_per_correct_problem
map

@playfulMIT
How to join expert and LA flows?
Evidence
Constructsmap
Data Features
data schema inform
algorithms
expert
assessment
ML/AI

@playfulMIT
Activity!
1. Use the constructs you defined during previous activity and add new extra ones.
For each construct:
a. Map the Evidence you found in the game, into the actual data that you need
b. Write human-readable instructions or pseudo-code that explains how to
process that data to generate features that represent a construct
c. What changes would you do in the game to facilitate capturing useful data to
measure that construct?

@playfulMIT
Share out
- Describe your game: What are the main mechanics? What learning is involved?
- For each constructs that you have identified as measurable:
a. Name the construct and describe it
b. What is the evidence you find in the game related to the construct?
c. How did you map the evidence into data?
d. How did you measure your constructs? Both ECD and LA are valid
e. Do you recommend a change in the game mechanics to improve the evidence
that the game can generate?
- What did you learn with this activity?

@playfulMIT
Key takeaways
- We applied ECD and LA approaches to measure constructs in games
- Very simplified version of the real complete process of game-based assessment,
we missed to talk about:
a. Initial design phase: domain knowledge-game mechanics-data alignment
b. Playtesting and iteration
c. Application and user target
d. Evaluation

Learning Analytics Design in Game-based Learning

More Related Content

Similar to Learning Analytics Design in Game-based Learning (20)

More from MIT (6)

Recently uploaded (20)

Learning Analytics Design in Game-based Learning

Editor's Notes