Towards Discovering the Role of Emotions in Stack Overflow
Download as PPT, PDF3 likes2,259 views
The document explores the impact of emotions on the success of questions and perceived quality of answers on Stack Overflow, using sentiment analysis and various metrics to assess emotional styles. It discusses preliminary findings that suggest a correlation between emotional tone, question success, and perceived author reputation. Future work aims to improve sentiment detection and further investigate the relationship between emotions and various aspects of user interactions in the platform.
![Problems in detecting
sentiment
ļ¬ āProblemā lexicon is too peculiar for the domain
to be considered as a pure expression of
negative emotions
ļ¬ Actually describing emotions
ļ¬ āI have very simple and stupid trouble [ā¦] I'm pretty
confused, explain please, what is wrong?ā (neg=-2)
ļ¬ āSorry for troubling you guysā (neg=-2)
ļ¬ Simply describing problem
ļ¬ What is the best way to kill a critical process? (neg=-2)
ļ¬ What is wrong? (neg=-2)
ļ¬ Mixed
ļ¬ Iām missing a parenthesis . But where? :( (neg=-3)
24](https://image.slidesharecdn.com/sseqaworkshop-141120002157-conversion-gate02/85/Towards-Discovering-the-Role-of-Emotions-in-Stack-Overflow-24-320.jpg)
Towards Discovering the Role of Emotions in Stack Overflow
- 1. Towards Discovering the Role of Emotions in Stack Overflow N. Novielli, F. Calefato, F. Lanubile University of Bari, Italy {nicole.novielli, fabio.calefato, filippo.lanubile}@uniba.it
- 2. A new way to access knowledge SSE@FSE 2014 2
- 3. How Do Programmers Ask and Answers Questions? ļ¬ Which questions are answered well and which ones remain unanswered? (Treude et al., ICSEā11), (Asudazzaman et al., MSRā13) ļ¬ Can we predict how long a question will remain unanswered? (Asudazzaman et al., MSRā13) ļ¬ What are the main discussion topics? (Barua et al., ā12), (Bajaji et al., MSRā14) ļ¬ What are the main factors affecting reputation? (Bosu et al., MSRā13)
- 4. Emotions in Social Computing and SSE ļ¬ Sentiment Analysis on Yahoo! Answers (Kucuktunc et al., WSDMā12) ļ¬ Answers perceived as good have a more neutral sentiment than others ļ¬ Do developers feel emotions? (Murgia, et al., MSRā14) ļ¬ Apache Software Foundation issue tracker ļ¬ Sentiment Analysis of Commit comments in GitHub (Guzman et al., MSRā13) ļ¬ Correlation with day and time, programming language, team distribution SSE@FSE 2014 4
- 5. Research Question Getting emotional while asking or answering questions in Stack Overflow: good or bad? ļ¬ Impact on success of questions ļ¬ Impact on perceived quality of answers ļ¬ Correlation with reputation ļ¬ Correlation with topics ļ¬ ā¦ SSE@FSE 2014 5
- 6. Preliminary study ļ¬ RQ1:To what degree does the emotional style of a question affect the probability of success? ļ¬ A successful question has an accepted answer SSE@FSE 2014 6
- 8. Dataset distribution SSE@FSE 2014 8 No accepted Answers (31%) No Answers (11%) Accepted Answers (58%) Successful 4,196,125 questions Unsuccessful 3,013,677 questions
- 9. Building the Model SSE@FSE 2014 9 Post Properties ā¢ Title Length ā¢ Post Length ā¢ Code Blocks ā¢ Day ā¢ Time ā¢ Topic ā¢ # Comments Social Factors ā¢ Question Score ā¢ Answer Score ā¢ # Accepted answer provided ā¢ # Answers accepted ā¢ # Badges Affective Factors ā¢Sentiment Polarity ā¢ Polarity of Question/Answer ā¢ Polarity of Comments ā¢Lexical Cues of Affective States ā¢ Positive emotions lexicon ā¢ Negative emotions lexicon ā¢ Gratitude ā¢ Politeness ā¢ Attitude of doubt ā¢ ā¦ Control Model
- 10. The Model Post Properties Social Factors Affective Factors SSE@FSE 2014 10 Control Model Independent variables, logistic regression model Dependent variable: success of a question (Y/N)
- 11. Post Properties - Metrics ā¢ Title and Post Length: # words ā¢ Alhoff at al., @ICWSMā14; Asaduzzaman et al., @MSRā13 ā¢ Used by SO moderators for automatic filtering ā¢ Code Blocks: yes/no ā¢ Treude et al., @ICSEā11 ā¢ Day: in {weekday, weekend} ā¢ Bosu et al., @MSR2013 ā¢ Time: in {morning, afternoon, evening night} ā¢ Bosu et al., @MSR2013 ā¢ Topic: categorical, using LDA ā¢ Asaduzzaman et al., @MSRā13; Bosu et al., @MSRā13 ā¢ Harper et al., @CHIā08 ā¢ Barua et al., Empirical Software Engineering 2014 SSE@FSE 2014 11
- 12. Social Factors - Metrics ā¢ Assessing the reputation of the author of the question at the time it is posted ā¢ High status correlated with success in Reddit.com (Althoff et al., ICWSMā14) ā¢ Novicesā questions are more likely answered on Stack Overflow (Treude et al., ICSEā 11) ā¢ Metrics to approximate the authorās reputation ā¢ Question Score: upvotes - downvotes on questions ā¢ Answer Score: upvotes ā downvotes on answers ā¢ # Accepted answer provided ā¢ # Answers accepted ā¢ # Badges: total badges owned SSE@FSE 2014 12
- 13. Affective Factors ā¢ Sentiment Polarity ā¢ Questions/Answers ā¢ Polarity of Comments SSE@FSE 2014 13
- 14. Sentiment Analysis Emotion Detection Subjective vs. Objective Negative vs. Positive Classification using Discrete Emotion Labels Goal āI can't solve this problem, itās very frustratingā SSE@FSE 2014 14 Example Resources - SentiStrength (Thelwall et al., 2012) - SentiWordNet (Esuli and Sebastiani, 2006) - MPQA Lexicon (Wilson et al., EMNLPā05) - ā¦ - LIWC (Tausczik and Pennebaker, 2010) - WordNet Affect (Strapparava and Valitutti, 2004) - Depeche Mood (Staiano and Guerini, ACLā14) - ā¦ Sad, Frustrated āI can't solve this problem, itās very frustratingā Subjective, Negative
- 15. Affective Factors ā¢ Sentiment Polarity ā¢ Question ā¢ Polarity of Comments ā¢ Lexical Cues of Affective States ā¢ Positive emotions lexicon ā¢ Negative emotions lexicon ā¢ Gratitude ā¢ Politeness ā¢ Attitude of doubt ā¢ ā¦ Future work - Sentistrength: http://sentistrength.wlv.ac.uk/ SSE@FSE 2014 15
- 16. SentiStrength ļ¬ Estimates the strength of both positive and negative sentiment in questions and comments ļ¬ Robust also for informal language ļ¬ Used in previous research ļ¬ Sentiment Analysis of commit comments in GitHub (Guzman et al., MSRā13) ļ¬ Sentiment Analysis on Yahoo! Answers (Kucuktnc et al., WSDMā12) SSE@FSE 2014 16
- 17. Preliminary results - Post Properties 17 Coeff Odds Ratio Code Blocks 0.2549 1.29 # of comments -0.3659 0.69 Day (Weekend) 0.0131 1.01 TIME Afternoon 0.1418 1.15 Evening 0.2093 1.23 Night 0.1085 1.12 Post LENGTH Body Length -0,0004 0.99 Title Length -0.0039 0.99 All significant, with a = 0.05 ā¢ Review questions are more concrete and get more answers (Treude et al., ICSEā11) and vague questions remain unanswered (Asaduzzaman et al., MSRā13) ā¢ SO off-peak hours (night): longer answer interval and less questions posted (Barua et al., MSRā13)
- 18. Post properties: Topic 18 Coeff Odds Ratio DATABASES/PERFORMANCE 0.4062 1.50 WEB PROGRAMMING 0.2725 1.31 GRAPHICS 0.2415 1.27 WEB PROGRAMMING/HTTP 0.1441 1.16 JAVA 0.0029 1.00 OOP 0.8599 2.36 MOBILE DEVELOPMENT/iOS 0.2664 1.30 SOURCE CODE MANAGEMENT 0.2805 1.32 DATA STRUCTURE/ALGORITHMS 0.7340 2.08 .NET FRAMEWORK/ASP 0.3442 1.41 SCRIPTING 0.3649 1.44 DATABASES/SQL 0.4488 1.57 WEB APP DEVELOPMENT 0.3330 1.40 MOBILE DEV/ANDROID 0.1111 1.12 All significant, with a = 0.05
- 19. Success rate per topic 19 Topic Success rate Number of questions Post rate OOP 6 70,81% 630258 8,84% DATA STRUCTURE/ALGORITHMS 9 67,73% 798713 11,20% DATABASES/SQL 12 61,12% 582130 8,16% .NET FRAMEWORK/ASP 10 58,73% 518834 7,28% SCRIPTING 11 58,54% 497763 6,98% WEB APP DEVELOPMENT 13 58,47% 492173 6,90% DATABASES/PERFORMANCE 0 57,72% 415825 5,83% WEB PROGRAMMING 1 56,59% 536255 7,52% SOURCE CODE MANAGEMENT 8 55,37% 373397 5,24% GRAPHICS 2 54,37% 383376 5,38% MOBILE DEVELOPMENT/iOS 7 53,91% 376517 5,28% WEB PROGRAMMING/HTTP 3 52,22% 375510 5,27% MOBILE DEV/ANDROID 14 51,50% 432095 6,06% JAVA 5 49,35% 235489 3,30% WEB AUTHENTICATION/API 4 49,00% 482992 6,77%
- 20. Preliminary Results ā Social Factors Coeff Odds Ratio User Question Score* -0,0017 0.99 User Answer Score* -0,0002 0.99 User Answers Accepted* 0,0047 1.00 User Questions Accepted* 0,0078 1.00 Number Of Badges 0,0001 1.0001103 SSE@FSE 2014 20 *significant with a = 0.05
- 21. Preliminary Results ā Affective Factors Coef Odds Ratio SENTIMENT of the QUESTION Question Positive Score -0.0248 0.98 Question Negative Score -0.0083 0.99 SENTIMENT of the authorās COMMENTS Comment Positive Score -0.1813 0.83 Comment Negative Score -0.1080 0.90 All significant, with a = 0.05 SSE@FSE 2014 21
- 22. Impact of Positive Sentiment on Success Positive polarity of QUESTION Positive polarity of COMMENTS 22
- 23. Impact of Negative Sentiment on Success Negative polarity of QUESTION Negative Polarity of COMMENTS 23
- 24. Problems in detecting sentiment ļ¬ āProblemā lexicon is too peculiar for the domain to be considered as a pure expression of negative emotions ļ¬ Actually describing emotions ļ¬ āI have very simple and stupid trouble [ā¦] I'm pretty confused, explain please, what is wrong?ā (neg=-2) ļ¬ āSorry for troubling you guysā (neg=-2) ļ¬ Simply describing problem ļ¬ What is the best way to kill a critical process? (neg=-2) ļ¬ What is wrong? (neg=-2) ļ¬ Mixed ļ¬ Iām missing a parenthesis . But where? :( (neg=-3) 24
- 25. - Thanks! Preliminary qualitative analysis using LIWC - Positive score = 3 SSE@FSE 2014 25
- 26. Next steps ļ¬ Separate positive emotions from gratitude expressions ļ¬ Qualitative analysis using of the first 1000 questions with highest positive sentiment score ļ¬ Gratitude and politeness are the most frequent cases ļ¬ āCheersā, āThanks (in advance)ā, āThank youā, ā¦ ļ¬ Gratitude is positively associated to success of requests (Althoff et al., 2014) 26
- 27. Next steps ļ¬ Further lexical analysis ļ¬ Assessing the suitability of state-of-the-art tools for sentiment analysis ļ¬ Modeling the āsuccess lexiconā ļ¬ Classification study: is success predictable? ļ¬ Preliminary results: 0.67 accuracy ļ¬ Investigate other research questions ļ¬ Emotions and perceived quality of answers ļ¬ Emotions and reputation ļ¬ Emotions and topics 27
- 29. Thank you N. Novielli, F. Calefato, F. Lanubile University of Bari, Italy {nicole.novielli, fabio.calefato, filippo.lanubile}@uniba.it
Editor's Notes
- #20: How do this relate with previous research on this domain? How do this relate with reputation and expert distribution in the Stack Overflow community?