“Ok Pal, we have to code that now”: Interaction Patterns of Programming Beginners with a Conversational Chatbot
Context: Chatbots based on large language models are becoming an important tool in modern software development, yet little is known about how programming beginners interact with this new technology to write code and acquire new knowledge. Thus, we are missing key ingredients to develop guidelines on how to adopt chatbots for becoming productive at programming.
Objective: With our research, we aim at identifying these ingredients. Specifically, we want to understand how programming beginners use conversational chatbots when writing source code. Method: To this end, we study programming beginners’ interaction with a chatbot in a CS2 course while they were solving programming assignments. Additionally, we evaluate the correctness of submitted solutions and compare them to solutions of beginners who did not use a conversational chatbot. Findings: We analyzed 756 prompts of 129 conversations, most of them focusing on code generation. Interestingly, conversations that contain prompts asking for debugging or testing of code are linked with higher success rates, indicating that deeper engagement with code leads to higher quality code. Moreover, prompts without sufficient context often lead to unsatisfying results. Implications: While not surprising, this underpins the importance that programming beginners need to know how to use chatbots, instead of merely using it as code generators without investing time in code quality. Moreover, companies should employ prompt guidelines, in which code quality prompts might be enforced after a code generation prompt has been stated.
Tue 24 JunDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
14:00 - 15:30 | ProcessIndustry Papers / Ideas, Visions and Reflections / Journal First / Research Papers at Aurora A Chair(s): Trey Woodlief University of Virginia, United States | ||
14:00 10mTalk | Non-Knowledge as a New Lens on Software Engineering Ideas, Visions and Reflections Jacob Krüger Eindhoven University of Technology, Xenia Marlene Zerweck Harz University of Applied Sciences, Sol Martinez Demarco Harz University of Applied Sciences, Alena Bleicher Harz University of Applied Sciences, Thomas Leich Harz University of Applied Sciences, Germany | ||
14:10 10mTalk | To Co-locate or Not to Co-locate? On the Impact of Hybrid Work to Software Design Process Ideas, Visions and Reflections Tommi Mikkonen University of Jyvaskyla, Mahum Adil Free University of Bozen/Bolzano, Ilenia Fronza Free University of Bozen/Bolzano, Italy, Gennaro Iaccarino Provincia Autonoma di Bolzano, Petri Ihantola University of Jyväskylä | ||
14:20 20mTalk | “Ok Pal, we have to code that now”: Interaction Patterns of Programming Beginners with a Conversational Chatbot Journal First Alina Mailach Leipzig University, Dominik Gorgosch Chemnitz University of Technology, Norbert Siegmund Leipzig University, Janet Siegmund Chemnitz University of Technology | ||
14:40 20mTalk | Revolutionizing Newcomers' Onboarding Process in OSS Communities: The Future AI Mentor Research Papers Xin Tan Beihang University, Xiao Long , Yinghao Zhu Beihang University, Lin Shi Beihang University, Xiaoli Lian Beihang University, China, Li Zhang Beihang University DOI Pre-print | ||
15:00 20mTalk | OurCode: Experiences Transitioning University Research into a Developer Tools Startup Industry Papers Consuelo Lopez OurCode Inc., Sahar Mehrpour George Mason University, USA, Austin Henley Carnegie Mellon University, Thomas LaToza George Mason University | ||
15:20 10mTalk | Polymer: Development Workflows as Software Ideas, Visions and Reflections Dhasarathy Parthasarathy Volvo Group, Yinan Yu Chalmers University of Technology, Earl T. Barr University College London |
Aurora A is the first room in the Aurora wing.
When facing the main Cosmos Hall, access to the Aurora wing is on the right, close to the side entrance of the hotel.