An Empirical Study on Developers' Shared Conversations with ChatGPT in GitHub Pull Requests and Issues
Thu 1 May 2025 13:00 - 13:30 at Canada Hall 3 Poster Area - Thu Lunch Posters 13:00-13:30
Fri 2 May 2025 10:30 - 11:00 at Canada Hall 3 Poster Area - Fri Morning Break Posters 10:30-11
ChatGPT has significantly impacted software development practices, providing substantial assistance to developers in various tasks, including coding, testing, and debugging. Despite its widespread adoption, the impact of ChatGPT as an assistant in collaborative coding remains largely unexplored. In this paper, we analyze a dataset of 210 and 370 developers’ shared conversations with ChatGPT in GitHub pull requests (PRs) and issues. We manually examined the content of the conversations and characterized the dynamics of the sharing behavior, i.e., understanding the rationale behind the sharing, identifying the locations where the conversations were shared, and determining the roles of the developers who shared them. Our main observations are: (1) Developers seek ChatGPT’s assistance across 16 types of software engineering inquiries. In both conversations shared in PRs and issues, the most frequently encountered inquiry categories include code generation, conceptual questions, how-to guides, issue resolution, and code review. (2) Developers frequently engage with ChatGPT via multi-turn conversations where each prompt can fulfill various roles, such as unveiling initial or new tasks, iterative follow-up, and prompt refinement. Multi-turn conversations account for 33.2% of the conversations shared in PRs and 36.9% in issues. (3) In collaborative coding, developers leverage shared conversations with ChatGPT to facilitate their role-specific contributions, whether as authors of PRs or issues, code reviewers, or collaborators on issues. Our work serves as the first step towards understanding the dynamics between developers and ChatGPT in collaborative software development and opens up new directions for future research on the topic.
Wed 30 AprDisplayed time zone: Eastern Time (US & Canada) change
16:00 - 17:30 | Human and Social Process 2Journal-first Papers / Research Track at 207 Chair(s): Armstrong Foundjem École Polytechnique de Montréal | ||
16:00 15mTalk | An Empirical Study on Developers' Shared Conversations with ChatGPT in GitHub Pull Requests and Issues Journal-first Papers Huizi Hao Queen's University, Canada, Kazi Amit Hasan Queen's University, Canada, Hong Qin Queen's University, Marcos Macedo Queen's University, Yuan Tian Queen's University, Kingston, Ontario, Ding Steven, H., H. Queen’s University at Kingston, Ahmed E. Hassan Queen’s University | ||
16:15 15mTalk | Who’s Pushing the Code: An Exploration of GitHub Impersonation Research Track Yueke Zhang Vanderbilt University, Anda Liang Vanderbilt University, Xiaohan Wang Vanderbilt University, Pamela J. Wisniewski Vanderbilt University, Fengwei Zhang Southern University of Science and Technology, Kevin Leach Vanderbilt University, Yu Huang Vanderbilt University | ||
16:30 15mTalk | Understanding Real-time Collaborative Programming: a Study of Visual Studio Live Share Journal-first Papers Xin Tan Beihang University, Xinyue Lv Beihang University, Jing Jiang Beihang University, Li Zhang Beihang University | ||
16:45 15mTalk | Characterizing the Prevalence, Distribution, and Duration of Stale Reviewer Recommendations Journal-first Papers Farshad Kazemi University of Waterloo, Maxime Lamothe Polytechnique Montreal, Shane McIntosh University of Waterloo | ||
17:00 15mTalk | Diversity's Double-Edged Sword: Analyzing Race's Effect on Remote Pair Programming Interactions Journal-first Papers | ||
17:15 7mTalk | Investigating the Impact of Interpersonal Challenges on Feeling Welcome in OSS Research Track Bianca Trinkenreich Colorado State University, Zixuan Feng Oregon State University, USA, Rudrajit Choudhuri Oregon State University, Marco Gerosa Northern Arizona University, Anita Sarma Oregon State University, Igor Steinmacher NAU RESHAPE LAB Pre-print |
Thu 1 MayDisplayed time zone: Eastern Time (US & Canada) change
13:00 - 13:30 | Thu Lunch Posters 13:00-13:30Research Track / SE in Society (SEIS) / Journal-first Papers / SE In Practice (SEIP) at Canada Hall 3 Poster Area | ||
13:00 30mTalk | BDefects4NN: A Backdoor Defect Database for Controlled Localization Studies in Neural Networks Research Track Yisong Xiao Beihang University, Aishan Liu Beihang University; Institute of Dataspace, Xinwei Zhang Beihang University, Tianyuan Zhang Beihang University, Li Tianlin NTU, Siyuan Liang National University of Singapore, Xianglong Liu Beihang University; Institute of Dataspace; Zhongguancun Laboratory, Yang Liu Nanyang Technological University, Dacheng Tao Nanyang Technological University | ||
13:00 30mTalk | Ethical Issues in Video Games: Insights from Reddit Discussions SE in Society (SEIS) | ||
13:00 30mTalk | An Empirical Study on Developers' Shared Conversations with ChatGPT in GitHub Pull Requests and Issues Journal-first Papers Huizi Hao Queen's University, Canada, Kazi Amit Hasan Queen's University, Canada, Hong Qin Queen's University, Marcos Macedo Queen's University, Yuan Tian Queen's University, Kingston, Ontario, Ding Steven, H., H. Queen’s University at Kingston, Ahmed E. Hassan Queen’s University | ||
13:00 30mTalk | QuanTest: Entanglement-Guided Testing of Quantum Neural Network SystemsQuantum Journal-first Papers Jinjing Shi Central South University, Zimeng Xiao Central South University, Heyuan Shi Central South University, Yu Jiang Tsinghua University, Xuelong LI China Telecom Link to publication | ||
13:00 30mPoster | FlatD: Protecting Deep Neural Network Program from Reversing Attacks SE In Practice (SEIP) Jinquan Zhang The Pennsylvania State University, Zihao Wang Penn State University, Pei Wang Independent Researcher, Rui Zhong Palo Alto Networks, Dinghao Wu Pennsylvania State University | ||
13:00 30mTalk | Building Domain-Specific Machine Learning Workflows: A Conceptual Framework for the State-of-the-PracticeSE for AI Journal-first Papers Bentley Oakes Polytechnique Montréal, Michalis Famelis Université de Montréal, Houari Sahraoui DIRO, Université de Montréal DOI Pre-print File Attached | ||
13:00 30mTalk | On the acceptance by code reviewers of candidate security patches suggested by Automated Program Repair tools.Security Journal-first Papers Aurora Papotti Vrije Universiteit Amsterdam, Ranindya Paramitha University of Trento, Fabio Massacci University of Trento; Vrije Universiteit Amsterdam | ||
13:00 30mTalk | Automating Explanation Need Management in App Reviews: A Case Study from the Navigation App Industry SE In Practice (SEIP) Martin Obaidi Leibniz Universität Hannover, Nicolas Voß Graphmasters GmbH, Hannah Deters Leibniz University Hannover, Jakob Droste Leibniz Universität Hannover, Marc Herrmann Leibniz University Hannover, Jannik Fischbach Netlight Consulting GmbH and fortiss GmbH, Kurt Schneider Leibniz Universität Hannover, Software Engineering Group |
Fri 2 MayDisplayed time zone: Eastern Time (US & Canada) change
10:30 - 11:00 | Fri Morning Break Posters 10:30-11Journal-first Papers / SE In Practice (SEIP) / Research Track / SE in Society (SEIS) / New Ideas and Emerging Results (NIER) at Canada Hall 3 Poster Area | ||
10:30 30mTalk | An Empirical Study on Developers' Shared Conversations with ChatGPT in GitHub Pull Requests and Issues Journal-first Papers Huizi Hao Queen's University, Canada, Kazi Amit Hasan Queen's University, Canada, Hong Qin Queen's University, Marcos Macedo Queen's University, Yuan Tian Queen's University, Kingston, Ontario, Ding Steven, H., H. Queen’s University at Kingston, Ahmed E. Hassan Queen’s University | ||
10:30 30mTalk | Automating Explanation Need Management in App Reviews: A Case Study from the Navigation App Industry SE In Practice (SEIP) Martin Obaidi Leibniz Universität Hannover, Nicolas Voß Graphmasters GmbH, Hannah Deters Leibniz University Hannover, Jakob Droste Leibniz Universität Hannover, Marc Herrmann Leibniz University Hannover, Jannik Fischbach Netlight Consulting GmbH and fortiss GmbH, Kurt Schneider Leibniz Universität Hannover, Software Engineering Group | ||
10:30 30mTalk | On the acceptance by code reviewers of candidate security patches suggested by Automated Program Repair tools.Security Journal-first Papers Aurora Papotti Vrije Universiteit Amsterdam, Ranindya Paramitha University of Trento, Fabio Massacci University of Trento; Vrije Universiteit Amsterdam | ||
10:30 30mTalk | Relevant information in TDD experiment reporting Journal-first Papers Fernando Uyaguari Instituto Superior Tecnológico Wissen, Silvia Teresita Acuña Castillo Universidad Autónoma de Madrid, John W. Castro Universidad de Atacama, Davide Fucci Blekinge Institute of Technology, Oscar Dieste Universidad Politécnica de Madrid, Sira Vegas Universidad Politecnica de Madrid | ||
10:30 30mTalk | BDefects4NN: A Backdoor Defect Database for Controlled Localization Studies in Neural Networks Research Track Yisong Xiao Beihang University, Aishan Liu Beihang University; Institute of Dataspace, Xinwei Zhang Beihang University, Tianyuan Zhang Beihang University, Li Tianlin NTU, Siyuan Liang National University of Singapore, Xianglong Liu Beihang University; Institute of Dataspace; Zhongguancun Laboratory, Yang Liu Nanyang Technological University, Dacheng Tao Nanyang Technological University | ||
10:30 30mTalk | Ethical Issues in Video Games: Insights from Reddit Discussions SE in Society (SEIS) | ||
10:30 30mTalk | SusDevOps: Promoting Sustainability to a First Principle in Software Delivery New Ideas and Emerging Results (NIER) Istvan David McMaster University / McMaster Centre for Software Certification (McSCert) |