ChatGPT: Friend or Foe When Comprehending and Changing Unfamiliar Code
A rapidly growing body of research is examining how LLMs influence developers when they code. To date, this research has tended to focus on productivity and code quality outcomes, rather than the underlying cognitive processes involved in programming. To address this gap, we report on the results of an exploratory laboratory study of ten advanced student developers (five with support from AI and five without) who had to make a non-trivial extension to a sizable software system. Leveraging Polya’s four problem-solving phases and 25 inductively-generated codes detailing distinct problem-solving behaviors as the primary lenses, we examined: (1) how AI impacted the problem-solving approach the developers used to solve the programming task, and (2) how AI impacted their progress when they became stuck. For the analysis, we triangulated data across multiple sources (e.g., think aloud, code changes, web searches, and LLM prompts). Unexpectedly, while developers in the AI group repeatedly turned to the AI tool to offload certain aspects of the process, all detailed problem-solving behaviors appeared in both groups. We also found that nine out of ten participants found themselves stuck in their work, but with key differences in how they became stuck and unstuck. We highlight seven distinct causes for being stuck and highlight how AI in some cases helped and in other cases hindered becoming unstuck.
Tue 9 JunDisplayed time zone: London change
15:30 - 17:00 | Human Factors in SE 1Research Papers / Journal First / Short Papers and Emerging Results at JMS 745 Chair(s): Mahmood Niazi King Fahd University of Petroleum and Minerals | ||
15:30 15mTalk | How do ML practitioners perceive explainability? an interview study of practices and challenges Journal First Umm-e Habiba TUM School of CIT, Technical University of Munich, Mohammad Kasra Habib TUM School of CIT, Technical University of Munich, Justus Bogner Vrije Universiteit Amsterdam, Jonas Fritzsch University of Stuttgart, Institute of Software Engineering, Stefan Wagner Technical University of Munich Link to publication | ||
15:45 15mTalk | Does social identity matter in software engineering? Assessing the case of research software engineers Research Papers Chukwudi Uwasomba The Open University, Tamara Lopez The Open University, Melanie Langer INSEAD, Helen Sharp The Open University, UK, Michel Wermelinger The Open University, Caroline Jay Department of Computer Science, University of Manchester, M13 9PL, United Kingdom, Mark Levine Lancaster University, Bashar Nuseibeh The Open University, UK; Lero, University of Limerick, Ireland Pre-print | ||
16:00 15mTalk | ChatGPT: Friend or Foe When Comprehending and Changing Unfamiliar Code Research Papers Norman Anderson University of Victoria, Tarek Alakmeh University of Zurich, Victoria Jackson University of Southampton, Guilherme Vaz Pereira School of Technology, PUCRS, Brazil, Umit Akirmak University of Victoria, Anthony Estey University of British Columbia, Rafael Prikladnicki School of Technology at PUCRS University, Thomas Fritz University of Zurich, Andre van der Hoek University of California, Irvine, Margaret-Anne Storey University of Victoria Pre-print | ||
16:15 15mTalk | Managing Power Gaps as an Element of Pair Programming Skill: A Grounded Theory Research Papers Linus Ververs Freie Universität Berlin, Janina Berger Freie Uinversität Berlin, Lutz Prechelt Freie Universität Berlin Pre-print | ||
16:30 10mTalk | Beyond Extraversion: Rethinking Personality-Based Explanations of Work Arrangement Preferences Short Papers and Emerging Results Darja Šmite Blekinge Institute of Technology, Eriks Klotins Blekinge Institute of Technology, Martin Svensson Blekinge Institute of Tecnology | ||