How heated is it? Understanding GitHub locked issues
Although issues are created to discuss and solve technical problems, conversations can get heated, with discussants getting angry and/or excited for a variety of reasons, such as poor suggestions or even, the violation of community conventions. To prevent and mitigate discussions from getting heated, communities like GitHub have introduced the ability to lock issue discussions that violate the code of conduct or other community guidelines. Despite some early research on locked issues, there is a lack of understanding of how communities use this feature and of potential threats to validity for researchers relying on a dataset of locked issues as an oracle for heated discussions. To address this gap, we (i) quantitatively analyzed 79 GitHub projects that have at least one issue locked as too heated, and (ii) qualitatively analyzed all issues locked as too heated of the 79 projects, a total of 205 issues and 5,511 comments. We found that projects have different behaviors when locking issues: 14 projects locked more than 90% of their closed issues, 54 locked less than 10% of their closed issues, and 11 locked between 54% and 88% of their closed issues. Additionally, locked issues tend to have more comments and more participants compared to non-locked issues. For the 205 issues locked as too heated, we found that one-third did not contain any uncivil discourse, and only 8.82% of the analyzed comments are actually uncivil. Finally, we found that the locking justifications provided by maintainers do not always match the label used to lock the issue. Based on our results, we identify three pitfalls to avoid when using the GitHub locked issues data.