Generative AI, notably ChatGPT, has garnered attention for its application in computer science education. This paper presents a controlled experiment that explores ChatGPT’s role in CS1 in a classroom setting. Specifically, we aim to investigate the impact of ChatGPT on student learning outcomes and their behaviors when working on programming assignments. Participants were tasked with creating a UML diagram and subsequently implementing its design through programming, followed by a closed-book post-evaluation and a post-survey. All the participants were required to screen-record the whole process. In total, 56 participants were initially recruited, with 48 successful screen recordings. Participants in the experimental group can access ChatGPT 3.5 and other online resources, such as Google and Stack Overflow when creating the UML diagram and programming; however, participants in the control group can access all online resources except for ChatGPT (i.e., the only design variable is the access to ChatGPT). Finally, we measured and analyzed participants’ learning outcomes through their UML diagram, programming, and post-evaluation scores. We also analyzed the time participants took to complete the tasks and their interactions with ChatGPT and other resources from the screen recordings. After finishing the tasks, student participants also provided their perceptions of using ChatGPT in CS1 through a post-survey.

With rigorous quantitative and qualitative analysis, we found that (1) using ChatGPT does not present a significant impact on students’ learning performance in the CS1 assignment-style tasks; (2) once using ChatGPT, students’ tendency to explore other traditional educational resources is largely reduced (though available) and they tend to rely solely on ChatGPT, and this reliance on ChatGPT did not guarantee enhanced learning performance; (3) the majority of students hold neutral views on ChatGPT’s role in CS1 programming but most of them raised concerns about its potential ethical issues and inconsistent performance across different tasks.

We hope this study can help educators and students better understand the impact of ChatGPT in CS1 and inspire future work to provide proper guidelines for using ChatGPT in introductory programming classes.