Detection of and response to Online Users' Emotion
Download as PPTX, PDF1 like243 views
The document discusses the importance of detecting and responding to users' emotions in online educational environments, highlighting how emotional engagement impacts learning. It examines various systems for personalized tutoring and patient care that utilize emotion detection through sensors and computer vision. Research findings suggest that incorporating empathy and emotional support can enhance learning outcomes and patient interactions.
Detection of and response to Online Users' Emotion
- 1. Beverly Park Woolf College of Information and Computer Sciences, University of Massachusetts-Amherst bev@cs.umass.edu Detection of and Response to Online Users’ Emotion Department of Quantitative Health Sciences University of Massachusetts Medical School March 24, 2017
- 2. Bounce between two systems . . . Mathematics Tutor: Exists and contains most of the described features. Patient Care Tutor: Proposed and can be built with these features.
- 3. Introduction and Care for the Patient Model the Patient Model Emotion (Sensors & Computer Vision) Model the Medical Domain Provide Interventions Assess Patient Learning Agenda
- 4. Caring: A Definition Caring implies actions based on another person’s wants and desires (Noddings, 1986). Entities that care consider the other person’s point of view, her needs and what she expects of us (Cooper, 2003). Caring includes the ability to empathize with others and take responsibility for their needs.
- 5. Caring Impacts Learning Findings from neuroscience suggest that all learning is affective in nature (Damasio, 1994). Students learn most effectively in a climate in which people care about them (Cooper, 2003). A major weakness of traditional psychology is to separate the intellect and affect (Vygotsky, 1986). Every person’s idea contains an affective attitude.
- 6. When Caring is Present . . Positive emotions and interactions create the ambience for learning and enable a student’s brain to remain curious and open (Cooper, 2002; 2003). A continuing, positive sense of self produces a constant positive feeling throughout the body, which leads to greater openness and willingness to engage in interactions (Winkley, 1996). Babies brains grow when they feel cared for.
- 7. When Caring is NOT Present . . . Students flounder in internal confusion (Cooper, 2003). Negative affect tends to produce a shutting down of one’s self, a withdrawal, stimulating protection and defense.
- 8. Research Questions about Emotion How are users’ emotions evidenced and measured? How do emotions predict learning? How accurate are emotion models (e.g., Markov or Bayesian Models) at predicting future emotions from student behaviors? How effective are interventions at responding to emotion??
- 9. Introduction and Care for the Patient Model the Patient Model Emotion (Sensors & Computer Vision) Model the Medical Domain Provide Interventions Propose Health Care Tutor Agenda
- 10. Model the Patient Model the Disease Personalize Tutoring Assess Learning Intelligent Tutoring Systems
- 11. Adele explains the importance of palpating the patient’s abdomen. Background Research: Shaw, Johnson , & Ganeshan (1999). In Proceedings of the third annual conference on Autonomous Agents (pp. 283-290). ACM.
- 12. Adele advising in a critical care scenario on the World Wide Web. Pedagogical Agents on the Web Erin Shaw, W. Lewis Johnson, and Rajaram Ganeshan
- 13. Adele instructs a student to answer a quiz when the student selects a urine dipstick test. Pedagogical Agents on the Web Erin Shaw, W. Lewis Johnson, and Rajaram Ganeshan
- 14. Background Research: Taking the Time to Care: Empowering Low Health Literacy Hospital Patients with Virtual Nurse Agents Timothy W. Bickmore, Laura M. Pfeifer College of Computer & Information Science Northeastern University {bickmore,laurap}@ccs.neu.edu • Ninety million Americans have inadequate health literacy, resulting in a reduced ability to read and follow directions in the healthcare environment. • Animated, empathic virtual nurse interface for educating and counseling hospital patients with inadequate health literacy in their hospital beds at the time of discharge. • Results indicate that hospital patients with low health literacy found the system easy to use, reported high levels of satisfaction, and most said they preferred receiving the discharge information from the agent over their doctor or
- 15. Patient Interacting with Virtual Nurse Taking the Time to Care Timothy W. Bickmore, Laura M. Pfeifer Chi 2009
- 16. Sample dialogue - relational aspects highlighted Taking the Time to Care Timothy W. Bickmore, Laura M. Pfeifer Chi 2009
- 17. Taking the Time to Care Timothy W. Bickmore, Laura M. Pfeifer Chi 2009
- 18. Self-report Ratings of the Virtual Nurse (mean (SD)) Taking the Time to Care Timothy W. Bickmore, Laura M. Pfeifer Chi 2009
- 19. Introduction and Care for the Patient Model the Patient Model Emotion (Sensors & Computer Vision) Model the Medical Domain Provide Interventions Propose Health Care Tutor Agenda
- 20. MathSpring An intelligent tutor for mathematics; Contains hundreds of math problem, grades 5-10; Used by tens of thousands of students; Aligned with Common Core standards; Detects and responds to student emotion; Improves student passing of state standardized tests; Uses multimedia characters to support student emotion; Built by Woolf & Arroyo at UMass-Amherst.
- 21. 23
- 23. Pedagogical Agents Adult Users Require Adult Companions Help to develop a positive emotional environment; Agents offers advice and encouragement. Empathize with patient Agents express full sentences of cognitive, meta-cognitive and emotional feedback. Build patients’ self-esteem and self-worth Agents are gendered and multicultural: White, Hispanic, African-American
- 24. Mass Statewide Standardized Tests Passing grades for experimental (dark grey) and control group (light grey); same grade, same school and same teacher.
- 25. Students represented by the yellow/green polygon used the Math Tutor; those represented by the blue polygon did not. The distribution of students using the tutor is shifted towards the right, towards more proficiency and above proficiency. Students were matched in terms of teacher; all were 7th grade students.
- 26. Introduction and Care for the Patient Model the Patient Model Emotion (Sensors & Computer Vision) Model the Medical Domain Provide Interventions Propose Health Care Tutor Agenda
- 27. Sensors to Detect Student Emotion David Cooper, Ph.D. Dissertation
- 28. Sensors (Clockwise): mental state camera, skin conductance bracelet, pressure sensitive mouse, pressure sensitive chair.
- 29. 31 Use Sensors
- 30. 32 The Students Rural-Area High School in MA (35 students) Geometry and Algebra classes UMASS 114 (29 students) Math for Elementary School Teachers
- 31. 33 Students Self-Report Emotions Four bipolar emotional axes Ekman’s Categorization Cognitive-Affective Term Emotion Scale High excitement Joy Low excitement “I am very excited.” . . . “This is not fun.” Frustration Anger Low-frustration “I am very frustrated.” . . “I am not frustrated at all.” High interest Interest Low interest “I am very interested.” . . . “I am bored.” Anxiety Fear Confidence “I feel anxious.” .. . . “I feel confident.” Table 1. Cognitive-affective terms based on human face studies (Ekman et al., 1972; Ekman 1999)
- 32. How are you feeling? Please rate your level of interest in this
- 33. Using Sensors to Measure Emotion: A Linear Regression Model Tutor context only Camera + Tutor Seat + Tutor Wrist + Tutor Mouse + Tutor Confident R=0.49, N=62 R=0.72, N=20 -- a R=0.35, N=32 R=0.55, N=28 Frustrated R=0.53, N=69 R=0.63, N=25 R=0.68, N=25 -- d R=0.56, N=45 R=0.54, N=44 Excite d R=0.43, N=66 R=0.83, N=21 -- b R=0.65, N=39 R=0.42, N=37 R=0.57, N=37 Interested R=0.37, N=94 R=0.54, N=36 -- c R=0.28, N=51 R=0.33, N=51 a --( Solved? ConcentratingMax); b -- (concentratingMax; unsureMin; HintsSeen; LearningCompanion?); c -- (InterestedMin LearningCompanion?); d -- (IncAttempts; SecsToFirstAtt; TimeInSession; sitForwardStdev R represents the fit of the model N is the number of cases available, for each emotion self-report and each sensor Significant results Student self-report Cooper et al., UMAP 2009; Arroyo et al., AIED 2009 David Cooper, Ph.D. Dissertation
- 34. 36 Models of Emotions with Sensors From Tutor-Context Variables and Sensors Linear Models to Predict Emotions Variables Entered in Stepwise Regression Confidence InterestFrustration Excitement SitForward Stdev “Concentrating” Max. Probability Camera Facial Detection Software SitForward Mean Seat Sensor # Hints Seen Solved? 1st Attempt # Incorrect attempts Gender Ped. Agent Seconds to 1st Attempt Time in Tutor Seconds To Solve Tutor Context Variables (for the last problem)Tutor Only All Sensors+Tutor R=0.53 R=0.43 R=0.37R=0.49 R=0.72 R=0.70 R=NAR=0.82 Sensor Variables (Mean, Min, Max, Stdev for the last problem)
- 35. Results of Emotion Prediction Studies Sensors predict self-report, accuracy 75-80%. Sensors improve student emotion self-reports. Self-reports predicts post-tutor math performance, attitudes and perceptions of tutoring software.
- 36. Introduction and Care for the Patient Model the Patient Model Emotion (Sensors & Computer Vision) Model the Medical Domain Provide Interventions Propose Health Care Tutor Agenda
- 37. Computer Vision to Monitor & Measure Grit • Several written Instruments are used to measure non-cognitive skills, primarily self-report surveys. – Self-regulation – Motivation strategies (MSLQ) • Students with grit & persistence have good self- regulation skills: – Goal setting, self-monitoring, and self-instruction; – They manage their emotions and waning motivation
- 38. Head Position The right mouth corner is tracked during student- computer interactions with significant changes of head position and orientation
- 39. Low Level Features Extraction of low-level features from face, (a) furrows, (b) iris, and (c) mouth, for FACS-based facial expression analysis, yielding the interpretation “student exhibits anger.”
- 40. Trace Student Attention The Mathspring GUI overlaid with a “heat map” of student attention, as measured by computer-vision. In red, screen regions the student particularly focused on
- 41. Engaged while reading and solving the problem
- 42. Head tilts
- 43. Starting to get annoyed with the learning companion • Student was actually fine with the learning companion at first but seems to get annoyed later on • Seem to be offended when the learning companion states “See, I told you that hints are really helpful”. He never took any hints and did everything correctly on every first try • Finally closes the learning companion as soon as it pops up after doing six Looking to the right side of the screen means he is looking at learning companion
- 44. Closing learning companion Companion removed by student.
- 45. Summary for GritCamera6 • Student is very engaged • Student takes time to answer every question – Made no mistake – Took no hint – Got it right in every first try • Takes time to answer survey – Only two survey shows up
- 46. Introduction and Care for the Patient Model the Patient Model Emotion (Sensors & Computer Vision) Model the Medical Domain Provide Interventions Propose Health Care Tutor Agenda
- 47. Cardiac Tutor, Built at UMass-Amherst The Cardiac Tutor teaches advanced cardiac support techniques to medical personnel. It presents cardiac problems and, using a variety of steps, students select various interventions. It provides clues, verbal advice, and feedback to personalize and optimize the learning.. Simulated patient in the Cardiac Tutor. The intravenous line has been installed ( “IV in ”); chest compressions are in progress, ventilation has not yet begun, and the electronic shock system is discharged. Eliot, Williams & Woolf, 1996, An intelligent learning environment for advanced cardiac life support.
- 48. Rashi Case Description Built at UMass-Amherst Dragon et al., 2006 Coaching Within a Domain Independent Inquiry Environment
- 49. Rashi Interview Tool Dragon et al., 2006 Coaching Within a Domain Independent Inquiry Environment
- 50. Rashi Exam Tool Dragon et al., 2006 Coaching Within a Domain Independent Inquiry Environment
- 51. Rashi Lab Test Tool
- 52. Rashi Inquiry Notebook Dragon et al., 2006 Coaching Within a Domain Independent Inquiry Environment
- 53. Rashi Argument Editor Dragon et al., 2006 Coaching Within a Domain Independent Inquiry Environment
- 54. Background Research: Medical Knowledge Bioworld patient scenario and evidence palette. Susanne Lajoie 2001 Constructing knowledge in the context of BioWorld
- 55. Model Medical Knowledge (Left) Conducting Diagnostic Tests; (Right) constructing the closing argument Lajoie 2001; Constructing knowledge in the context of BioWorld Lajoie & Greer, 2003 Establishing an Argumentation Environment to Foster Scientific Reasoning With Bio-World
- 56. Introduction and Care for the Patient Model the Patient Model Emotion (Sensors & Computer Vision) Model the Medical Domain Provide Interventions Propose Health Care Tutor Agenda
- 57. Research Questions • What to do in the moment when users are frustrated, bored, etc.? – Increase Challenge? Decrease Challenge? – Provide extra scaffolds? – Provide “affective” scaffolds? What are those? – Encourage users to stop and think about what is going on? • How to measure changes in student affect, or to capture micro-changes in student affective states?
- 58. Interventions have a strong impact on Learning • Average effect size of metacognitive instruction across 20 studies was .72, a very large effect. – Teach a skill; track the student, measure the skill before and after use of the software. – Measure a student’s aptitude for learners trained in a skill against students who are not trained in that skill.
- 59. Many people: • relate to computers in the same way they relate to humans (Nass, 2010); • continue to engage in frustrating tasks significantly longer after an empathic digital response (Picard); • have lowered stress levels after receiving an empathetic message from a digital character (Arroyo et al., 2009); • recalled more information when interacting with an artist agent compared to scientist agent; • report reduced frustration and more general interest when working with gender-matched characters. People and Agents
- 60. Affirmation Theory Messages Affirmation Theory: Propose that users’ motivation is rooted in their belief about why they succeed or fail. Students can be taught to understand that failure is the result of a lack of effort instead of a lack of ability. Example Messages “People have myths about math, like, that only some people are good in math. The truth is that we can all be successful in math if we give it a try” “We will learn new skills only if we are persistent. If we are very stuck, let's call the teacher, or ask for a hint!” “When we realize we don't know why the answer was wrong, it helps us understand better what we need to practice.”
- 61. Effort Interaction Messages Effort Interactions: Acknowledge effort and incorrect answers. The goal is to make students realize that praise is not always appropriate and that effort is the primary goal. Example Messages “That was too easy for you. Let's hope the next one is more challenging so that we can learn something.”!” “Good job! See how taking your time to work through these questions can make you get the right answer?”
- 62. Incorrect ResponseStudent effort shown/ correct response Student effort shown / incorrect response Agent Emotion Frustrated students are supported by helpful companions. Arroyo et al., AIED2009
- 63. • Our current Gold standard/Truth: self-reports • During Experiment: – Ask students about their “interest” level and their “excitement” level every 7 minutes, on average. Gathering Student Affect “During” Low High Neutral / Middle
- 64. Large Data Sets EventLog Table of a Math Tutoring System. 571,776 rows, just in a year time.
- 65. The Student Progress Page 73 Dovan Rai, PhD, WPI
- 66. Three Experimental Conditions 74 • No access to the Student Progress Page • “My Progress” button was present (student choice) • Prompt invitation to see “my progress” upon bored (disinterested) or unexcited • Force student to see my progress when student bored (disinterested) or unexcited
- 67. How to analyze changes in student affect, from moment to moment? MARKOV CHAIN MODELS
- 68. Student Interest Ten thousand Data Points N~230. SPP Absent SPP Present SPP Prompted SPP Forced
- 69. How to compare Markov Chain Models, quantitatively? • Probability of student following a specific path • What is the probability that a student will end up excited, after 3 transitions? • Given that they started in a specific state?
- 70. Resulting Data
- 71. Introduction and Care for the Patient Model the Patient Model Emotion (Sensors & Computer Vision) Model the Medical Domain Provide Interventions Propose Health Care Tutor Agenda
- 72. Health Care Research Questions Which interventions improve patients’ affect and learning? Which interventions are best for which patients? What is the impact of deeper companion characteristics? Do affective characters make users “feel” better? How does the gender (ethnicity) of companions impact patients’ attitudes/emotions/learning?
- 73. Detect patient emotion, using Computer Vision or sensors (camera, wrist band, pressure mouse, seat sensors). Dynamically collect data streams of students’ physiological activity and self-reports of emotions. Apply interventions once emotion is detected, e.g., animated embodied agents Integrate computer vision techniques to improve detection of emotion. Train researchers and educators in techniques to recognize an respond to patients’ emotion and make predictions over larg data sets. Proposed Health Care System
- 74. Study Design Patients Control Experimental Intervention Control Experimental Intervention Visit # 1 Visit # 2 Visit # 3 Visit # 4 Randomized • Survey • Audio tape with physician • Survey • Audio tape with physician • Survey • Audio tape with physician • Survey • Survey • Survey • Survey Baseline data Greenfield et al., 1986; Patients’ participation in Medical Care General Internal Medicine
- 75. Emotion is vital to learning and using technology to recognize users’ emotion has led to powerful performance results. Detecting and responding to users’ emotion while they work with online environments improves learning. Summary
- 76. Thank You ! Any Questions? Detection of and Response to Online Users’ Emotion Department of Quantitative Health Sciences University of Massachusetts Medical School March 24, 2017