Using Pattern Matching to Assess Gameplay
Download as PPTX, PDF1 like760 views
The document discusses the use of Analysis of Patterns in Time (APT) to assess learner interactions within serious games, particularly highlighting the Diffusion Simulation Game. It compares APT to traditional measurement methods, showcasing how APT can create temporal maps to analyze gameplay data, enabling formative assessments. The research also includes a case study, demonstrating APT's application and implications for understanding gameplay dynamics and outcomes.
Using Pattern Matching to Assess Gameplay
- 1. USING PATTERN MATCHING TO ASSESS GAMEPLAY RODNEY D. MYERS, PH.D. INDEPENDENT SCHOLAR THEODORE W. FRICK, PH.D. PROFESSOR EMERITUS, INDIANA UNIVERSITY 1
- 2. INTRODUCTION Using Analysis of Patterns in Time (APT) to measure and analyze a learner’s interactions with a serious game. • Overview of MAPSAT and APT • Comparison with traditional methods • Examples and explanations • Case Study • The Diffusion Simulation Game and diffusion of innovations theory • Using APT to analyze gameplay data • Using APT for Formative Assessment • Concluding Remarks 2
- 3. OVERVIEW OF MAPSAT Map & Analyze Patterns & Structures Across Time: 2 methods • Analysis of Patterns in Time (APT) • Analysis of Patterns in Configuration (APC) APT • Different approach to measurement and analysis • Create a temporal map which characterizes temporal events • Look for temporal patterns within a map • Count them (event pattern frequency) • Estimate likelihood (relative frequency) • Aggregate time (event pattern duration) • Estimate proportion time (relative pattern duration) 3
- 4. HOW IS APT DIFFERENT? • Traditional quantitative methods of measurement and analysis • Obtain separate measures of variables for each case • Statistically analyze relations among measures • We relate measures Example of spreadsheet data: 4
- 5. HOW IS APT DIFFERENT? • Analysis of Patterns in Time • Create temporal map for each case • Query temporal map for patterns • We measure relations directly Example of spreadsheet data: 5 Map Query 1 Query 2 1 0.30 0.58 2 0.25 0.67 3 0.40 0.56
- 6. WHAT IS A TEMPORAL MAP? 6 Example of temporal map of weather JTE Unix Epoch Time Started: Duration of JTE Season of Year Air Temperature (degrees F) Barometric Pressure (p.s.i.) Precipitation Cloud Structure 1 1417436508: dur. = 1470 { Fall { 33 { Above 30 { Null { Cirrus 2 1417437978: dur. = 2277 | | { Below 30 | | 3 1417440255: dur. = 2554 | | | | { Nimbus Stratus 4 1417442809: dur. = 794 | | | { Rain | 5 1417443603: dur. = 1095 | { 32 | | | 6 1417444698: dur. = 477 | | | { Sleet |
- 7. CODEBOOK FOR OBSERVING WEATHER EVENTS Classification 0 Name: Season of Year Classification Value Type = Nominal Number of categories (temporal event values) = 5 Category 0 = Null Category 1 = Fall Category 2 = Winter Category 3 = Spring Category 4 = Summer Classification 1 Name: Air Temperature Classification Value Type = Interval Units of measure = degrees Fahrenheit 7
- 8. CODEBOOK FOR OBSERVING WEATHER EVENTS Classification 2 Name: Barometric Pressure Classification Value Type = Ordinal Number of categories (temporal event values) = 3 Category 0 = Null Category 1 = Above 30 psi Category 2 = Below 30 psi Classification 3 Name: Precipitation Classification Value Type = Nominal Number of categories (temporal event values) = 4 Category 0 = Null Category 1 = Rain Category 2 = Sleet Category 3 = Snow 8
- 9. QUERY A TEMPORAL MAP: EXAMPLE Query 1. Here is a 2-phrase APT Query: WHILE the FIRST Joint Temporal Event is true (Phrase 1): Season of Year is in state starting or continuing, value = Fall Barometric Pressure is in state starting or continuing, value = Below 30 Cloud Structure is in state starting or continuing, value = Nimbus Stratus • Duration when Phrase 1 is True = 13,436 seconds (out of 19,584 seconds total). Proportion of Time = 0.68607 • Joint Event Frequency when Phrase 1 is True = 12 (out of 18 total joint temporal events). Proportion of JTEs = 0.66667 THEN while the NEXT Joint Temporal Event is true (Phrase 2): Season of Year is in state starting or continuing, value = Fall Barometric Pressure is in state starting or continuing, value = Below 30 Precipitation is in state starting or continuing, value = Rain Cloud Structure is in state starting or continuing, value = Nimbus Stratus • Duration when Phrase 2 is True = 4,086 seconds (out of 19,584 seconds total), given all prior phrases are true. Proportion of Time = 0.20864 • Joint Event Frequency when Phrase 2 is True = 3 (out of 18 total joint temporal events), given all prior phrases are true. Proportion of JTEs = 0.16667 • Conditional joint event duration when Phrase 2 is true, given all prior phrases are true = 0.30411 (4,086 out of 13,436 seconds (time units). • Conditional joint event frequency when Phrase 2 is true, given all prior phrases are true = 0.25000 (3 out of 12 joint temporal events). 9
- 10. RESULT OF QUERY FOR APT PATTERN IN TEMPORAL MAP 10 The conditional joint event duration of the 2-phrase pattern specified in Query 1 becomes the measure that is entered into the spreadsheet Map Query 1 Query 2 1 0.30 0.58 2 0.25 0.67 3 0.40 0.56 Thus, the variable is the pattern specified Query 1 and its value is 0.30.
- 11. DEMO OF APT QUERIES ON WEATHER PATTERNS If we have a good Internet connection: https://www.indiana.edu/~simed/aptdemo/aptdsg.php 11
- 12. EXAMPLE OF APT TEMPORAL MAP FOR THE DIFFUSION SIMULATION GAME If we have a good Internet connection: https://www.indiana.edu/~simed/aptmultimap/aptdsg.php 12
- 13. THE DIFFUSION SIMULATION GAME & DOI THEORY 13 Case Study: Using APT for Serious Games Analytics Diffusion Simulation Game (DSG)
- 14. USING APT TO ANALYZE GAMEPLAY DATA Generalizations from DOI theory 14 DSG activities Adopter types Decision phases Example: Mass media should be effective in spreading knowledge about an innovation, especially among innovators and early adopters Local Mass Media & Print Innovators & Early Adopters Awareness & Interest 9 strategies: patterns of joint occurrences file:///Users/tedfrick/Documents/AECT%202014/View%20Temporal%20Map% 20DSG%20MultiMap.html
- 15. USING APT TO ANALYZE GAMEPLAY DATA • Revised DSG to require login to track changes in gameplay over time • Reviewed “finished” games • 109 players finished 1 or more games • 27 players finished 2 or more games • 14 players finished 3 or more games • Selected 3 players to serve as examples 15
- 16. GAME OUTCOMES 16 Game Outcome Adoption Points Maximally Successful 220 Highly Successful 166 – 219 Moderately Successful 146 – 165 Unsuccessful 0 - 145 Game Outcomes Player 1 Un Un Un Md Player 2 Un Un Md Hi Md Hi Md Un Hi Hi Hi Player 3 Un Hi Md Hi Mx Hi
- 17. EXAMPLE QUERY 17 Query Result for Player 1, Game 3 WHILE the FIRST Joint Temporal Event is true (Phrase 1): Diffusion Activity is in state starting or continuing, value = Local Mass Media Turn Rank is in state starting or continuing, value <= 3 • Duration when Phrase 1 is True = 1 moves (out of 59 DSG moves total). Proportion of Time = 0.02222 • Joint Event Frequency when Phrase 1 is True = 1 (out of 74 total joint temporal events). Proportion of JTEs = 0.01351
- 18. EXAMPLE QUERY 18 Use of Local Mass Media activity by game outcome and strategy rank for turn. Player 1 Un Un Un Md Overall 0.00 0.00 0.03 0.03 High 0.00 0.00 0.02 0.03 Low 0.00 0.00 0.02 0.00 Player 2 Un Un Md Hi Md Hi Md Un Hi Hi Hi Overall 0.00 0.05 0.03 0.07 0.05 0.06 0.04 0.05 0.02 0.10 0.10 High 0.00 0.00 0.00 0.02 0.02 0.02 0.02 0.00 0.00 0.05 0.05 Low 0.00 0.00 0.00 0.00 0.02 0.00 0.00 0.02 0.00 0.02 0.02 Player 3 Un Hi Md Hi Mx Hi Overall 0.02 0.07 0.10 0.10 0.05 0.09 High 0.02 0.02 0.03 0.03 0.03 0.03 Low 0.00 0.05 0.05 0.08 0.03 0.06
- 19. USING APT FOR FORMATIVE ASSESSMENT DURING GAMEPLAY 19 • Summative • Used by instructor and/or learner • Evidence of understanding and application • Formative • Dynamic analysis of gameplay • Provide scaffolds (e.g., hints, coaching) • Requested by learner • Before turn: hint • After turn: explanation or prompt for reflection • Analyze prior gameplay maps • Identify persistent misconceptions
- 20. USING APT FOR FORMATIVE ASSESSMENT DURING GAMEPLAY 20 Player 3 Un Hi Md Hi Mx Hi Overall 0.02 0.07 0.10 0.10 0.05 0.09 High 0.02 0.02 0.03 0.03 0.03 0.03 Low 0.00 0.05 0.05 0.08 0.03 0.06 Poor use of mass media Generalization 5-13: Mass media channels are relatively more important at the knowledge stage, and interpersonal channels are relatively more important at the persuasion stage in the innovation-decision process (p. 205). Generalization 7-22: Earlier adopters have greater exposure to mass media communication channels than do later adopters (p. 291).
- 21. CONCLUDING REMARKS 21 “Using Pattern Matching to Assess Gameplay” to be published in: Loh, C. S., Sheng, Y., & Ifenthaler, D. (Eds.). (2015). Serious game analytics: Methodologies for performance measurement, assessment, and improvement. New York, NY: Springer. Contact us: Rod Myers – rod@webgrok.com Ted Frick – frick@indiana.edu
Editor's Notes
- #14: To test the effectiveness of APT for serious games analytics, we used it to analyze gameplay data from the Diffusion Simulation Game. The DSG is an online game in which a player takes on the role of a change agent whose task is to influence the principal and teachers at a junior high school to adopt peer tutoring. The player gathers information about the staff and tries to engage them in appropriate activities to raise their awareness and interest and eventually adopt peer tutoring. To be successful in the game, the player must apply concepts from the diffusion of innovations theory, including adopter types, phases of the innovation-decision process, and the importance of social networks.
- #15: To specify performance indicators, we identified generalizations from Rogers’ book Diffusion of Innovations that were applicable while playing the DSG. We mapped these statements to actions that may be taken in the DSG, which involve combinations of activities, adopter types, and innovation-decision phases. For example…. Next we identified data associated with these actions and designed a database for data collection in which the columns are APT event classifications (e.g., activity selected, current stage in the innovation-decision process for each staff member) and the rows contain the relevant categories in each classification for each turn in a game. We identified nine strategies from DOI theory that are applicable in the DSG, and we wrote a strategy scoring algorithm that analyzed the game state for each turn and assigned a score to each strategy based on the likelihood of its success in that turn. This approach enabled us to compare what the player did during each turn with what the player should have done based on DOI theory.