Affective Metacognitive Scaffolding and User Model Augmentation for Experiental Training Simulators: A Follow-up Study
Download as PPTX, PDF1 like1,035 views
The document discusses a study on augmenting experiential training simulators with affective metacognitive scaffolding and user model enhancements to improve adult learning experiences. It examines the effectiveness of the ETU medical interview training simulator, analyzing factors such as relevance, user integration, and self-regulated learning outcomes across two user trials. Findings indicate improvements in user preparedness, motivation, and overall learning experience, as well as challenges related to user acceptance of data sharing.
Affective Metacognitive Scaffolding and User Model Augmentation for Experiental Training Simulators: A Follow-up Study
- 1. Affective Metacognitive Scaffolding and User Model Augmentation for Experiental Training Simulators: A Follow-up Study Immersive Reflective Experience-based Adaptive Learning 8th European Conference on Technology Enhanced Learning (EC-TEL 2013) Gudrun Wesiak, Adam Moore, Christina M. Steiner, Claudia Hauff, Conor Gaffney, Declan Dagger, Dietrich Albert, Fionn Kelly, Gary Donohoe, Gordon Power, & Owen Conlan
- 2. Overview - Intro - ImREAL - Challenges and Approach - ETU Simulator - Simulator Augmentation - Empirical study - Methods and results - Discussion & Conclusions EC-TEL 2013 2
- 3. Intro - Experiential Training Simulators - For engaging and motivating learning experiences - To create situational contexts to practice job-relevant skills - Adult learning (Andragogy) - Self-directed, experienced-based, goal-oriented, intrinsically motivating, relevancy-driven (Knowles, 1984) - Influenced by learner’s physical, digital and social world - Should adapt to learner’s previous experiences, knowledge, …. EC-TEL 2013 3
- 4. 4 ImREAL: Challenge Simulated environment Augment existing training simulators by... Real-world activity modelling Enriched user modelling Affective-metacognitive scaffolding service Develop services for different simulators Real-world experience Close the gap Medical interview training Fig. 2. Example learner interaction screen – the simulated situation is in the Dinner episode where the host has to decide about ordering food for his business guests. The simulated scenario includes four episodes: Greetings (situations embed arriving on time, different norms about greetings, first impression, and use of body language), Dinner (situations embed use of body language and different preferences about food Business simulation & Buddy program Simulated world Real world Forethought Reflection
- 5. Experiential Training Simulator: EmpowerTheUser - ETU Medical Interview Training: Two Psychiatric Interview Scenarios Mania & Depression Basic idea: Watch patient video and select appropriate answer 2Training modes: Assess mode and practice mode EC-TEL 2013 5
- 6. ETU: Assess mode Performance Prediction and Scores EC-TEL 2013 6 Interview steps (Process) Interview skills ETU scores
- 7. ETU: Practice mode EC-TEL 2013 7 Visible Decision scores Coach: feedback on decision path
- 8. Thinking prompts and scaffolding service ETU: Practice mode Meta-cognitive scaffolding (MSS) EC-TEL 2013 8
- 9. ETU: Practice mode Meta-cognitive scaffolding (MSS) EC-TEL 2013 9 Prompts are based on MAI-ROC (Schraw, 194) Prompts are mapped to different interview situations
- 10. ETU: Practice mode: Affective Metacognitive Scaffolding EC-TEL 2013 10 SAM: Smiley Affect Measurement technology • Explicit assessment of current emotional state with SAM • Selection of displayed prompts influenced by users’ emotional state • Sentiment analysis of digital traces (U-SEM) • Mappings of prompts to situations as weighted list
- 11. Empirical Study: Research Questions • Simulator Augmentation: • Is the simulation relevant for users? • Are the services well integrated? • Digital Traces: • Usage, privacy & trust EC-TEL 2013 12 • AMSS: Are there effect on… • self-regulated learning? • users’ learning experience? • learning performance? • motivation/affect?
- 12. Longterm Methodology: Berthold et al., EC-TEL 2012 Participants UT1: 143 TCD medical students, 22.7 years, 54.8% male, 97% real world interview experience (15.4% psychiatric) UT2: 152 TCD medical students, 22.81 years, 50% male, 99% real-world interview experience (25.6% psychiatric) • 2011 • Pure Simulator Baseline • 2012 • ETU with MSS User Trial 1 • 2013 • AMSS, SA M, U-SEM User Trial 2 Methods EC-TEL 2013 13
- 13. Methods: Procedure and Measures EC-TEL 2013 14 Cohort survey (N=95) Training in the simulator (N=143/152) Post-sim survey (N=39/ 40) Real-world interview Post-int survey (N=7/19) • Demographics • Social Network use & attitudes • Log-data (Usage, scores) 2 scenarios, 2 modes • Real-World Relevance • SRL-strategies • Motivation & Affect • Learning Experience • Learning Performance • Real-world relevance • Motivation • Fb on modes, AMSS, SAM, SN
- 14. Results: Log Data Simulator Usage & Training Times 15 Usage (solid lines) and mean training times (dashed) of assess and practice mode UT1: very few users for Mania UT2: short times for Depression Assess vs. Practice: Assess: more users and longer times UT1 vs. UT2: Same no. of users, but increased overall training times (both modes)
- 15. Results Which aspects are (not) effected? No differences in ... Integration & perception of thinking prompts Self-regulated learning strategies (QSRL) Motivation to improve skills, to perform good, to apply what has been learned (post-sim & post-int) No difference between UT1 & UT2, but change from Baseline Perceived relevance of simulation Usability (SUS) Overall motivation Overall workload (decreased!) Deacrease from UT1 to UT2 Motivation to learn more about interviewing (post-sim) EC-TEL 2013 16 Baseline UT1/UT2 UT1=UT2
- 16. Results Relevance for Real-World Interviews UT2 students feel better prepared for real-world interviews Perception of relevancy & preparedness unchanged after real-world interview 17
- 17. Results Learning Experience Flow (Flow short scale; Rheinberg et al., 2003) increase in overall flow, technical flow & 7 single items i.e. stronger involvement into the task, greater feeling of smoothness and fluency, smooth interplay between software components 18 Flow score means on item basis (left fig.) and overall means (and SE) for the FKS, worry, and tech scales (right fig.)
- 18. Results Learning Experience Workload: Nasa-TLX (Hart & Staveland,1988) Lower overall workload in UT2 UT2 users felt less frustrated & had stronger feelings of success EC-TEL 2013 19 Mean scores (and SE) from the NASA-TX workload scales for the two user trials.
- 19. Results Self-Regulated Learning Reflection notes and AMSS prompts • 1092 triggers provided (M=15/student) in UT2, 2001 in UT1 • MAI-prompts: 88.4% of 69 answers positive • Increase in proportion of reflective notes EC-TEL 2013 20 Table 1. Rounded percentages of content types for entries from the note-taking tools. Note: Entries can be coded to more than one type, thus percentages may exceed 100. Users Text entries Position Technical Notes Reflection UT1 50 107 17 57 16 66 UT2 35 86 1 13 7 93
- 20. Results: Log Data ETU Scores ETU: predicted and achieved scores in UT1 and UT2 21 Higher scores in UT2 for introduction, eliciting and planning phase Mainly underestimation of scores in UT2
- 21. Results Affect Affective state reported after simulation: Total affective state scales (TASS, Nicolescu et al., 2006 ) Affect during simulation: SAM was displayed 352 times (2.23 times / user) – used by 1 person 22 Mean TASS scores on 7 subscales plus overall mean score across all scales. NO Yes
- 22. Results Social Networks Usage: 81% use facebook, 20% twitter, < 5% Flickr, MySpace, linkedIn Nobody (of 152) provided SN-ID in simulator (13 did in cohort survey; 11 fb) For detailed results: see Moore, 2013 (Umap; LBR) Reasons: EC-TEL 2013
- 23. Results Summary • Changes from pure (BL) to augmented (UT) simulator ↑ Perceived relevance, usability, state motivation ↓ Overall workload • Changes from MSS (UT1) to AMSS (UT2) simulator Relevancy: ↑ stronger feeling of preparedness SRL: ↑ effort (times), feeling of success, reflective notes; Performance: ↑ ETU-Scores Learning Experience: ↑ flow ↓ decrease in frustration Motivation: ↓ to learn more about interviewing EC-TEL 2013 24
- 24. Conclusions & Challenges • Metacognitive scaffolding supports learners on different levels • Evaluation of user model augmentation Low user acceptance for exploiting their social digital traces Need to increase willingness to share SN-IDs • Study learning activity rather than purely subjective reports Analysing learners’ real activities with the simulators • (Non)-invasiveness vs. (un)-visibility SAM, thinking prompts EC-TEL 2013 25
- 25. Conclusions & Challenges • (Non)-invasiveness vs. (un)-visibility SAM now with triggers and presented in main window EC-TEL 2013 26 NO YES
- 26. Thanks for listening! EC-TEL 2013 27