Ec tel2015: In-app Reflection Guidance for Workplace Learning
0 likes96 views
The document discusses the concept of in-app reflection guidance designed to enhance reflective learning in workplace settings. It outlines the methodology of user studies conducted with three different applications, assessing the perceived usefulness of reflection components and the levels of reflective learning achieved. Findings indicate that effective prompts and contextualization are essential for facilitating reflective practices among users.
Ec tel2015: In-app Reflection Guidance for Workplace Learning
- 1. b b © Know-Center GmbH, www.know-center.at In-App Reflection Guidance for Workplace Learning Angela Fessl, Gudrun Wesiak, Verónica Rivera-Pelayo, Sandra Feyertag, Viktoria Pammer EC-TEL 2015, Toledo, 17.09.2015
- 2. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics Introduction • In-App reflection guidance concept to trigger reflective learning • … at the workplace 2 Purpose of this work
- 3. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics Introduction • Conscious re-evaluation of past situations/experiences with the goal to learn from them & use the gained outcomes to guide future behaviour • Workplace learning: core process to get new insights, derive better practices -> improve own work • Cognitive process based on intrinsic & extrinsic motivation of an individual 3 Reflective Learning
- 4. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics Background – Reflection Guidance • RL Tools: Prompts, Reflection Amplifiers, Diaries, Journals, e- Portfolios • Prompting approaches successful in formal learning environments • Journals/Diaries in educational settings e.g. nursing, athletics • Little research yet in work-related settings • Challenges: • Work tasks not always known beforehand/vaguely known • Right time to display prompts? • Working environment is often stressful & work-intensive 4
- 5. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics Reflection Guidance Concept • Reflection interventions • Prompts to motivate users to utilise an application in general • At beginning/end of day, at the start of an event or when application was not used for a longer time • Reflection amplifiers • Well-considered prompts initiate reflection & support planning and reflection • Make users aware of unusual behaviour/working patterns or other significant situations • Motivate users to insert their thoughts into corresponding application 5 Reflection-in-Action Components (Verpoorten et al., 2010)
- 6. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics Reflection Guidance Concept • Reports • Automatic summary of experiences/events captured during work • Hourly, daily, weekly • Compare them with own or other‘s working behaviour detect patterns & insights learning outcomes • Reflection Diary • Store insights, thoughts, artefacts, learning outcomes in structured way • Re-evaluation of entries trigger reflective learning & lead to insights guiding future behaviour 6 Reflection-on-Action Components
- 7. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics Reflection Guidance Concept • Contextualisation • Manually or automatically tracked by application • Automatic tracking: extensive amounts of data • Reflective learning support: • User has to think about context information when entering it • Context information when revisiting captured data helps remembering the respective working situation better 7 Interwined Component
- 8. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics Methodology User studies in real work settings (3 companies, 3 Apps with reflection guidance components) 1. How were the implemented reflection guidance components perceived with regard to their usefulness at work? 2. To what extent (at which stage) did reflective learning occur? 8 Research Questions
- 9. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics Methodology • Log-Data objective usage rates • Pre-Questionnaire: demographic data • Post-Questionnaire • Support of apps with respect to RL • Perceived usefulness of reflection guidance components • Interviews & workshop: deeper information 9 Analysis Data
- 10. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics Methodology • Analysis of written content: qualitative analysis schema (independent analysis & categorisation by 3 researchers) • 1. stage: descriptions of experiences & emotions • 2. stage: interpretation & justification of actions and working on solutions • 3. stage: insights, learning & conclusions from reflection 10 Evaluations (Prilla et al., 2014)
- 11. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics User Studies • MoodMap App (Call Center) • Insights on how mood influences work performance & collaboration within a team • Medical Quiz (Stroke Unit) • Refresh existing knowledge & gain new knowledge • Connect theoretical knowledge with working practices & reflection • KnowSelf (IT Company) • Support of individual reflective learning regarding time management & self-organisation of knowledge workers 11 Overview and Learning Goals
- 12. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics User Study: MoodMap App at Call Center • Maps mood on a coloured map along 2 dimensions: • Valence (feeling good – feeling bad) • Arousal (high energy – low energy) • Reflection Guidance: • Contextualisation: For each mood choose pre-defined context & add note (supported by reflection amplifier question, e.g. What currently influences your mood?) • Reports (mood development over day) & reflection diary 12 MoodMap App
- 13. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics User Study: MoodMap App at Call Center 13 MoodMap App
- 14. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics User Study: MoodMap App at Call Center • Call Center: • 4 teams of 2 call centers of a large British telecommunication company • Each team: • Call takers (take calls & solve issues directly with customers) • Coaches (support & coach call takers for their work) • Managers (supervice training sessions, review call takers‘ performance & ensure their performance against targets) • 4 weeks • All participants were asked to integrate MoodMap App in their daily work routines 14 MoodMap App – Setting
- 15. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics User Study: Medical Quiz at Hospital • Quiz especially developed for nurses at a Stroke Unit in German hospitals • 4 different quiz types • Randomised content-based questions • Reflection guidance: reflection amplifiers • At the beginning: • address current knowledge status and play frequency • In-between: • put user‘s focus on content-based questions • At the end: • ask for gained insights or new knowledge w.r.t. the played quiz 15 Medical Quiz
- 16. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics User Study: Medical Quiz at Hospital 16 Medical Quiz
- 17. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics User Study: Medical Quiz at Hospital • Qualification program for becoming nurses working for a Stroke Unit • 5 months • Education: 1 week per month • Nurses were asked to play quiz consequently • Strengthen newly gained knowledge • Reflect about content & draw connections between new knowledge and daily work practices 17 Medical Quiz – Setting & Evaluation Procedure
- 18. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics User Study: KnowSelf at IT Company • Automatic tracking of resources & apps used on PC • Manual project & task recording • Reflection Guidance: • Reflection diary for notes, thoughts & insights • Time-triggered prompts • Remind user of reflecting by offering feedback on different aspects of user‘s behaviour • Event-triggered prompts • After detection of a significant change (e.g. unusually high number of switches) 18 KnowSelf
- 19. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics User Study: KnowSelf at IT Company 19 KnowSelf
- 20. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics User Study: KnowSelf at IT Company • Consulting Company • Consults, sells & personalises CRM (Customer Relationship Management) Software • Reflection regarding interactions with customers • Installation on business PCs • Reflect on collected data on daily basis over 6 weeks • Weekly reminder • Reflect on captured data • Enter observations • Fill in short questionnaire 20 KnowSelf - Setting
- 21. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics Results • MoodMap App: 33 participants • Interaction: M=42.67 (SD=72.2) from max. 338 to min 1 • Reflective Learning: M=3.25 (SD = .92) • Medical Quiz: 18 participants • Questions answered: M = 461.9 (SD=341), from max 1358 to min. 25 • Reflective Learning: M = 3.51 (SD=.42) • KnowSelf: 10 participants • Usages: M=38.5 min (SD 23.8), from max. 53.72min to min. 7.76min • Reflective Learning: M = 3.28 (SD =.66) • Sufficient app usage to evaluate reflection guidance components • Components perceived as useful and reflective learning has taken place • Positive correlations between extent of app usage & users‘ perceived support for reflective learning (significant for Medical Quiz & KnowSelf) 21 Did participants use apps, and did they actually reflect?
- 22. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics Results: Usage and Usefulness of Reflection Guidance • Context distribution: • After a call: 31% • After a break: 6% • After a coaching session: 2% • Other: 61% • How helpful was the contextualisation for • Associating inserted moods to certain situations: M=3.39 (SD=.86) • Recalling past experiences: M=3.21 (SD=.74) • Helping to better reflect about past situations: M=3.24 (SD=.94) • Context should be adaptable to individual preferences 22 Contextualisation – MoodMap App
- 23. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics Results: Usage and Usefulness of RGC • Time-triggered prompts • Reminder to use app: M=3.22 (SD=1.09) • Reminder of project recording: M=2.75 (SD=1.28) 23 Reflection Interventions - KnowSelf
- 24. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics Results: Usage and Usefulness of RGC • MoodMap App: Add note to moods • 475 notes inserted • Medical Quiz: reflective questions • 1205 posed, 603 (52%) meaningful answered • 59.3% at beginning, 51.5% at end, 42.4% in-between • Willingness to reflect is higher at beginning and end of the quiz than inbetween (disruptive) • KnowSelf: event-triggered prompts • Notification about most used resources: M=3.30 (SD=1.25) • Notification about unusual amount of idle time: M=2.57 (SD=0.79) • Notification about specific amount of switches: M=2.30 (SD=1.34) • Asking the right questions at the right time can trigger reflective learning 24 Reflection Amplifiers: MoodMap App, KnowSelf, Medical Quiz
- 25. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics Results: Content Analysis of inserted Notes • MMA: 457 notes to 548 inserted moods: 283 individual reflective items • 95% 1.stage (descriptions of experiences & emotions) • 5% 2.stage (interpretations/justifications of actions & working on solutions) • KnowSelf: 103 statements 33 non-reflective, 11 no inter-coder agreement, 59 reflective items • 22% 1.stage (descriptions of experiences & emotions) • 47% 2.stage (interpretations/justifications of actions & working on solutions) • 27% 3.stage (insights, learning outcomes, conclusions from reflection) • Medical Quiz: Answers to open questions were too short for analysis (often only a single word) 25 Stages of reflection
- 26. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics Conclusion 1. How were the implemented reflection guidance components perceived with regard to their usefulness at work? • Components perceived as useful • In-app reflection guidance facilitates technology-supported reflective learning at work 2. To what extent (at which stage) did reflective learning occur? • Achieved reflection stages differed between the apps and setting • MoodMap App: • 1. stage (experience) & 2. stage (interpretation) • KnowSelf • 1. stage (experience) & 2. stage (interpretation) & 3. stage (insights) 26 Research Questions
- 27. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics Conclusion • Correct timing of reflection interventions & amplifiers is crucial -> avoid disturbing interruptions • Interruptibility: ongoing research challenge • Give users control over notification 27 Lessons Learned
- 28. © Know-Center GmbH gefördert durch das Programm COMET (Competence Centers for Excellent Technologies), wir danken unseren Fördergebern: bQUESTIONS Angela Fessl afessl@know-center.at Know-Center GmbH Inffeldgasse 13 8010 Graz 28
- 29. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics References • Prilla, M., Renner, B.: Supporting collaborative reflection at work: a comparative case analysis. In: Proceedings of ACM Conference on Group Work (GROUP 2014). ACM (2014) • Schön, D.A.: The Reflective Practitioner: How Professionals Think In Action, 1st edn. Basic Books, New York (1984) • Verpoorten, D., Westera, W., Specht, M.: Reflection amplifiers in online courses: a classification framework. J. Interact. Learn. Res. 21(4), 654–666 (2010). 29