ICSME 2024
Sun 6 - Fri 11 October 2024

This program is tentative and subject to change.

Large Language Models (LLMs) have taken the world by storm, demonstrating their ability not only to automate tedious tasks, but also to show some degree of proficiency in completing software engineering tasks. A key concern with LLMs is their “black-box” nature, which obscures their internal workings and could lead to societal biases in their outputs. In the software engineering context, in this early results paper, we empirically explore how well LLMs can automate recruitment tasks for a geographically diverse software team. We use OpenAI’s ChatGPT to conduct an initial set of experiments using GitHub User Profiles from four regions to recruit a six-person software development team, analyzing a total of 3,657 profiles over a five-year period (2019–2023). Results indicate that ChatGPT shows preference for some regions over others, even when swapping the location strings of two profiles (counterfactuals). Furthermore, ChatGPT was more likely to assign certain developer roles to users from a specific country, revealing an implicit bias. Overall, this study reveals insights into the inner workings of LLMs and has implications for mitigating such societal biases in these models.

This program is tentative and subject to change.

Fri 11 Oct

Displayed time zone: Mountain Time (US & Canada) change

13:30 - 15:00
Session 15: Developer Experience and CommunicationIndustry Track / Research Track / Journal First Track / New Ideas and Emerging Results Track at Fremont
Chair(s): Alexander Serebrenik Eindhoven University of Technology
13:30
25m
Overcoming the five fundamental challenges to enable “constant velocity indefinitely” in modern software systemsIndustry Track Talk
Industry Track
Doug Durham Don't Panic Labs
13:55
15m
Investigating Developers' Preferences for Learning and Issue Resolution Resources in the ChatGPT EraDistinguished Paper AwardResearch Track Paper
Research Track
14:10
15m
Remote Communication Trends Among Developers and Testers in Post-Pandemic Work EnvironmentsIndustry Track Paper
Industry Track
Felipe Jansen CESAR School, Ronnie de Souza Santos University of Calgary
Link to publication Pre-print
14:25
10m
Nigerian Software Engineer or American Data Scientist? GitHub Profile Recruitment Bias in Large Language ModelsNIER Paper
New Ideas and Emerging Results Track
Takashi Nakano Nara Institute of Science and Technology, Kazumasa Shimari Nara Institute of Science and Technology, Raula Gaikovina Kula Nara Institute of Science and Technology, Christoph Treude Singapore Management University, Marc Cheong the University of Melbourne, Kenichi Matsumoto Nara Institute of Science and Technology
Pre-print
14:35
10m
Cross-status Communication and Project Outcomes in OSS Development– A Language Style Matching PerspectiveJ1C2 PaperVideo presentation
Journal First Track
Yisi Han Nanjing University, Zhendong Wang University of California, Irvine, Yang Feng Nanjing University, Zhihong Zhao Nanjing Tech Unniversity, Yi Wang Beijing University of Posts and Telecommunications