She Elicits Requirements and He Tests: Software Engineering Gender Bias in Large Language Models
Implicit gender bias in software development is a well-documented issue, such as the association of technical roles with men. To address this bias, it is important to understand it in more detail. This study uses data mining techniques to investigate the extent to which 56 tasks related to software development, such as assigning GitHub issues and testing, are affected by implicit gender bias embedded in large language models. We systematically translated each task from English into a genderless language and back, and investigated the pronouns associated with each task. Based on translating each task 100 times in different permutations, we identify a significant disparity in the gendered pronoun associations with different tasks. Specifically, when starting with the pronoun “she”, requirements elicitation was associated with the pronoun “he” in only 6% of cases, while testing was associated with “he” in 100% of cases. Additionally, tasks related to helping others had a 91% association with “he” while the same association for tasks related to asking coworkers was only 52%. These findings reveal a clear pattern of gender bias related to software development tasks and have important implications for addressing this issue both in the training of large language models and in broader society.
Tue 16 MayDisplayed time zone: Hobart change
14:35 - 15:15 | Human AspectsTechnical Papers / Data and Tool Showcase Track at Meeting Room 110 Chair(s): Alexander Serebrenik Eindhoven University of Technology | ||
14:35 12mTalk | A Study of Gender Discussions in Mobile Apps Technical Papers Mojtaba Shahin RMIT University, Australia, Mansooreh Zahedi The Univeristy of Melbourne, Hourieh Khalajzadeh Deakin University, Australia, Ali Rezaei Nasab Shiraz University Pre-print | ||
14:47 12mTalk | Tell Me Who Are You Talking to and I Will Tell You What Issues Need Your Skills Technical Papers Fabio Marcos De Abreu Santos Northern Arizona University, USA, Jacob Penney Northern Arizona University, João Felipe Pimentel Northern Arizona University, Igor Wiese Federal University of Technology, Igor Steinmacher Northern Arizona University, Marco Gerosa Northern Arizona University Pre-print | ||
14:59 6mTalk | She Elicits Requirements and He Tests: Software Engineering Gender Bias in Large Language Models Technical Papers Pre-print Media Attached | ||
15:05 6mTalk | GitHub OSS Governance File Dataset Data and Tool Showcase Track Yibo Yan University of California, Davis, Seth Frey University of California, Davis, Amy Zhang University of Washington, Seattle, Vladimir Filkov University of California at Davis, USA, Likang Yin University of California at Davis Pre-print |