Keynote@QUATIC - Recognizing Developer's Emotions: Advances and Open Challenges
0 likes50 views
The document discusses the recognition of developers' emotions in the context of software engineering, emphasizing the importance of emotional intelligence for productivity and wellbeing. It outlines various research studies and methodologies utilized for emotion detection, including biometrics such as EEG and GSR, and introduces research questions focusing on the range, triggers, and relationships between developers' emotions and productivity. It also highlights open challenges in the field and the need for further validation in diverse settings.
Keynote@QUATIC - Recognizing Developer's Emotions: Advances and Open Challenges
- 1. Recognizing Developers' Emotions: Advances and Open Challenges @NicoleNovielli nicole.novielli@uniba.it QUATIC - September 13, 2022 Nicole Novielli University of Bari, Italy Collaborative Development Group
- 2. Faculty • Filippo Lanubile • Nicole Novielli • Fabio Calefato Visiting Professors and Researchers COLLAB - Collaborative Development Group Research topics PhD Students • Luigi Quaranta • Daniela Grassi • Giuseppe Colavito • Serena Versino Final-year undergrad students
- 3. My research • Human Aspects in Software Engineering • Affective Computing • Natural Language Processing
- 4. Recognizing developers’ emotions BrainLink EEG monitors the electrical activity of the brain
- 5. Emotions in Software Development
- 6. Software Engineering involves social interaction • Programmers cooperate, directly or indirectly • Massive adoption of social media and rise of the ‘social programmer’ (Storey, ‘12) and the surrounding ecosystem • Computer-mediated interaction became prevalent during the pandemic!
- 7. Studying Emotions in Software Engineering
- 8. Studying Emotions in Software Engineering
- 9. Studying Emotions in Software Engineering
- 10. Collaborative software development and knowledge-sharing – Correlation of emotions with issue-fixing time (Ortu et al., MSR 2015) – Early burnout discovery (Mantyla et al. MSR 2015) – Anger detection (Gachechiladze et al., ICSE-NIER 2017) – Empirically-driven guidelines for question writings (Calefato et al., IST 2018) – Confusion in code reviews (Ebert et al., SANER 2019) – Negative sentiment in SADT (Fucci et al., MSR 2021) Recommender systems – Pattern-based mining of opinions in Q&A websites (Lin et al., ICSE 2019) – Opinion search and summarization for APIs (Uddin and Khomh, ASE 2017) Requirements engineering – User feedback (Guzman and Maalej, RE’14; van Oordt and Guzman, RE ’21, Kurtanovic and Maalej, 2020) – App improvement (Panichella et al., ICSME ‘14) Actionable insights for
- 11. Emotions and Wellbeing at the Workplace
- 12. Developers’ emotions, productivity and wellbeing
- 13. Developers’ emotions, productivity and wellbeing
- 14. Developers’ emotions, productivity and wellbeing
- 15. Developers’ emotions, productivity and wellbeing
- 16. Our vision
- 17. Biometrics
- 18. Acknowledgements Filippo Lanubile University of Bari Davide Fucci BTH Alexander Serebrenik TU/e Daniela Girardi University of Bari Daniela Grassi University of Bari
- 20. Electroencephalography (EEG) • Electrical activity of the brain • Cerebral waves categorized based on their frequency • Delta (<4 Hz): recorded during sleep • Theta (4-7,5 Hz): decrease of vigilance level • Alpha (4-12,5 Hz): relax • Beta (13-30 Hz): mental process • Gamma (>30 Hz): anxiety
- 21. Galvanic skin response (GSR) • Electrical activity of the skin • Changes due to the variation in body sweating • Electrical changes in the skin could be due to variation in emotions
- 22. Empatica E4
- 24. Circumplex Model of Affect J. A. Russell, A circumplex model of affect, Journal of Personality and Social Psychology 1980, Vol 39, no. 6
- 25. Lab studies
- 27. Controlled experiment Can we use noninvasive, low cost sensors for reliable emotion recognition? VS. https://www.eecs.qmul.ac.uk/mmv/datasets/deap/
- 28. Controlled experiment Machine learning - performance comparable with the state of the art
- 30. What is the minimal set of non-invasive biometric sensors to recognize developers’ emotions?
- 31. Previous studies
- 32. Instrumentation BrainLink EEG monitors the electrical activity of the brain
- 33. Instrumentation • 23 participants • Java development tasks • Gold standard: self report and arousal
- 34. Self-report of emotions and progress Bradley, M. M., & Lang, P. J. (1994). Measuring emotion: the self-assessment manikin and the semantic differential. Journal of behavior therapy and experimental psychiatry, 25(1), 49-59. Valence (Un)Pleasantness of the emotion stimulus Arousal Level of activation of the emotion stimulus
- 35. Gold standard based on self report Valence Arousal Posi ti ve Nega ti ve High Low 44 (32%) 94 (68%) 85 (62%) 53 (38%)
- 36. Machine Learning Preprocessing signals’ noise removal Feature extraction statistical measures of signals Classification Hold out -90-10 and Leave-one-subject-out (LOSO) setting https:// fi gshare.com/articles/_/9206474
- 37. Machine Learning
- 38. Sensor-based emotion detection Minimal set of sensors
- 40. Field study
- 41. Research questions RQ1. What is the range of developers’ emotions at the workplace? RQ2. To what extent are developers emotions related to self-assessed productivity during the workday? RQ3. What are the triggers for developers’ emotions at the workplace? RQ4. Can we predict developers’ emotions at the workplace using lightweight biometric sensors?
- 42. Research questions RQ1. What is the range of developers’ emotions at the workplace? RQ2. To what extent are developers emotions related to self-assessed productivity during the workday? RQ3. What are the triggers for developers’ emotions at the workplace? RQ4. Can we predict developers’ emotions at the workplace using lightweight biometric sensors? Biometrics
- 43. Link to full paper IEEE Transactions on Software Engineering, 2021
- 44. Study protocol 2/3 weeks
- 46. Productivity Self-assessed Values ranging from very low to very high
- 47. Triggers for self-reported emotions
- 48. . coding bug fi xing testing design meeting email, helping networking learning administrative task documentation just arrived other Based on Meyer et al. (TSE, 2019)
- 49. Participants Five companies Dutch software development companies, including - One startup (1 founder and 2 employees) - Two SMEs (between 20 and 200 employees) - Two large companies (> 20.000 employees) 21 professional developers - 18 men, 3 women - Average age: 33 years ± 7.2, ranging from 23 to 50 - Average experience in software development: 8 years ± 6.2, ranging from 1 to 25
- 51. Emotions and Productivity RQ1. What is the range of developers’ emotions at the workplace? RQ2. To what extent are developers emotions related to self-assessed productivity during the workday?
- 52. Developer’s emotion at the workplace
- 53. Developer’s emotion at the workplace
- 54. Valence is positively correlated with perceived productivity, with stronger correlation in the afternoon. Interaction between valence and time (day vs. afternoon) Emotions and self-assessed productivity
- 55. Valence is positively correlated with perceived productivity Stronger correlation in the afternoon. Conversely, the correlation between dominance and productivity is stronger in the morning. This could be due to fatigue, which is known to impair emotion regulation. Emotions and self-assessed productivity
- 56. Emotion Triggers
- 57. Emotion Triggers RQ3. What are the triggers for developers’ emotions at the workplace?
- 58. 48 pos: 18 neg: 19 RQ3: triggers of emotions
- 59. Triggers for positive and negative emotions
- 60. Triggers for positive and negative emotions
- 61. Triggers for positive and negative emotions
- 63. Can we predict developers’ emotions at the workplace using lightweight biometric sensors?
- 64. Minimal sensor setting for emotion recognition while programming
- 66. Machine Learning
- 67. Sensor-based emotion recognition Good: Our classi fi ers outperform the baseline
- 68. Sensor-based emotion recognition Good: The performance is comparable to the on in the lab
- 69. Sensor-based emotion recognition Expected: Hold-out scores are higher than leave-one-subject out
- 70. Variability between participants Variability between participants Also observed in the lab study (Girardi et al., ICSE 2020)
- 76. • Individual training/ fi ne-tuning of emo ti on classi fi ers • Further valida ti on with larger/more diverse pool of par ti cipants from di ff erent companies • Self-disclosure of nega ti ve emo ti ons • Analysis of GSR peaks as proxy of stress Open challenges and future studies
- 77. • Individual training/ fi ne-tuning of emo ti on classi fi ers • Further valida ti on with larger/more diverse pool of par ti cipants from di ff erent companies • Self-disclosure of nega ti ve emo ti ons • Analysis of GSR peaks as proxy of stress Open challenges and future studies
- 78. • Individual training/ fi ne-tuning of emo ti on classi fi ers • Further valida ti on with larger/more diverse pool of par ti cipants from di ff erent companies • Self-disclosure of nega ti ve emo ti ons • Analysis of GSR peaks as proxy of stress Open challenges and future studies
- 79. • Individual training/ fi ne-tuning of emo ti on classi fi ers • Further valida ti on with larger/more diverse pool of par ti cipants from di ff erent companies • Self-disclosure of nega ti ve emo ti ons • Analysis of GSR peaks as proxy of stress Open challenges and future studies
- 81. • Individual training/ fi ne-tuning of emo ti on classi fi ers • Further valida ti on with larger/more diverse pool of par ti cipants from di ff erent companies • Self-disclosure of nega ti ve emo ti ons • Analysis of GSR peaks as proxy of stress Open challenges and future studies
- 82. Emotions as a proxy for engagement in users’ interviews Wed Sept 2 2020