Incivility in Open Source Projects: A Comprehensive Annotated Dataset of Locked GitHub Issue Threads
0 likes7 views
This document presents a dataset of 404 locked GitHub issue threads, specifically curated to analyze incivility in open source software (OSS) discussions. The dataset includes 5,961 individual comments annotated for features of incivility, with common issues identified as bitter frustration, technical disagreements, and personal attacks. The findings aim to inform automated moderation solutions and enhance understanding of incivility impacts on OSS project health.
![● A curated dataset of 404 locked issue threads
from 213 GitHub projects [Scan QR Code]
● Bitter frustration, Impatience, and Mocking
are the most prevalent TBDFs
● Failed use of tool/code or error messages
the most common trigger
● People are the most common target
● Discontinued further discussion is the most
common consequence
Preprint: https://arxiv.org/abs/2307.15631
ramtin.ehsani@drexel.edu
Preprint: https://arxiv.org/abs/2402.04183
imranm3@vcu.edu
Summary Research Directions
● Automated moderation bot development
● Impact of incivility on project health
● Effectiveness of moderation strategies
● Early warning systems development
● Underrepresented communities'
experiences
● Predicting heated thread locking
● Identifying productive intervention points](https://image.slidesharecdn.com/msr24incivilitydatasetpresentation-240423174518-03c3bfcd/85/Incivility-in-Open-Source-Projects-A-Comprehensive-Annotated-Dataset-of-Locked-GitHub-Issue-Threads-6-320.jpg)
Incivility in Open Source Projects: A Comprehensive Annotated Dataset of Locked GitHub Issue Threads
- 1. 21st International Conference on Mining Software Repositories Incivility in Open Source Projects: A Comprehensive Annotated Dataset of Locked GitHub Issue Threads Ramtin Ehsani, Mia Mohammad Imran, Robert Zita, Kostadin Damevski, Preetha Chatterjee Drexel University Preprint: https://arxiv.org/abs/2402.04183 Virginia Commonwealth University Elmhurst University imranm3@vcu.edu
- 2. Motivation and Research Objective ● Fostering healthy collaborations in OSS is challenging ● Understanding and addressing incivility within OSS discussions ● A lack of a comprehensive approach to address uncivil interactions ● Lack of large annotated SE datasets Research Objective: Curating a dataset of locked GitHub issues enables analyzing incivility in OSS development Annotated dataset of locked GitHub issue threads with heated discussions
- 3. Dataset Annotation ● 404 Locked issue threads from 213 GitHub projects, and 5,961 Individual comments ● Locked as "too heated" or demonstrated clear characteristics indicative of heated discussions ● A total of 19 annotators ● To further improve the annotation quality, we used GPT-4 ● Manually checked the instances of disagreements between GPT-4 and annotators
- 4. ● Tone Bearing Discussion Features (TBDFs), uncivil features* ○ Bitter frustration, Impatience, Mocking, Irony, Vulgarity, etc ● Triggers* ○ Failed use of code, Technical disagreements, Communication breakdown, etc ● Targets* ○ People, Code/Tool, Company/organization, Undirected ● Consequences* ○ Discontinued further discussion, Escalating further, etc * C. Miller, S. Cohen, D. Klug, B. Vasilescu and C. Kästner, "“Did You Miss My Comment or What?” Understanding Toxicity in Open Source Discussions," 2022 * Isabella Ferreira, Jinghui Cheng, and Bram Adams, The "Shut the f**k up" Phenomenon: Characterizing Incivility in Open Source Code Review Discussions, 2021 * Jaydeb Sarker, Asif Kamal Turzo, Ming Dong, and Amiangshu Bosu, Automated Identification of Toxic Code Reviews Using ToxiCR, 2023 * Our open coding process Annotated Features
- 5. Dataset Description ● 1,365 comments annotated with an uncivil feature ● Bitter frustration, Impatience, and Mocking are the most prevalent TBDFs ● Failed use of tool/code or error messages the most common Trigger ● People are the most common Target ● Discontinued further discussion is the most common Consequence
- 6. ● A curated dataset of 404 locked issue threads from 213 GitHub projects [Scan QR Code] ● Bitter frustration, Impatience, and Mocking are the most prevalent TBDFs ● Failed use of tool/code or error messages the most common trigger ● People are the most common target ● Discontinued further discussion is the most common consequence Preprint: https://arxiv.org/abs/2307.15631 ramtin.ehsani@drexel.edu Preprint: https://arxiv.org/abs/2402.04183 imranm3@vcu.edu Summary Research Directions ● Automated moderation bot development ● Impact of incivility on project health ● Effectiveness of moderation strategies ● Early warning systems development ● Underrepresented communities' experiences ● Predicting heated thread locking ● Identifying productive intervention points