A Comparative Study on ChatGPT and Checklist as Support Tools for Unit Testing Education
Testing is widely practiced in software engineering, and many tools have been developed to support students in learning testing. Prior research suggests that a lightweight testing checklist improves learning outcomes but doesn’t address students’ challenges in writing test code that matches their intentions or design. Meanwhile, generative AI tools (e.g., ChatGPT) bring new promise as another form of software assistance tool. In this study, we examined the impact of various support tools (checklist, ChatGPT, or both) on unit testing among 42 students. Our results indicated that using these tools individually or in combination produced a comparable effect on student performance in unit testing. Students preferred using the checklist but acknowledged ChatGPT’s effectiveness in accelerating task completion and addressing programming language challenges. While ChatGPT demonstrated potential benefits for testing education, it did not overcome the implementation challenges identified in the previous study. Moreover, reliance on ChatGPT may hinder students’ deeper engagement with new concepts, which is crucial for comprehensive learning, as they often interacted superficially with AI-generated responses without employing the critical thinking necessary to evaluate the information provided. Therefore, we proposed recommendations for both students and instructors on adapting to learning and teaching in the AI era and offer insights into the evolving role of AI in education.
Tue 24 JunDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
14:00 - 15:30 | |||
14:00 20mTalk | Layered Learning: Teaching CI/CD and Software Testing through Storytelling Software Engineering Education | ||
14:20 20mTalk | Sojourner under Sabotage: A Serious Testing and Debugging Game Software Engineering Education Philipp Straubinger University of Passau, Tim Greller University of Passau, Gordon Fraser University of Passau | ||
14:40 20mTalk | Evaluating the Impact of Scaffolding and Visualizations for Mutation Testing Exercises in Software Engineering Education Software Engineering Education Hannah Potter University of Washington, Ana Paiva INESC TEC, Faculty of Engineering, University of Porto, Domenico Amalfitano University of Naples Federico II, Anna Rita Fasolino Federico II University of Naples, Porfirio Tramontana University of Naples Federico II, René Just University of Washington | ||
15:00 20mTalk | A Comparative Study on ChatGPT and Checklist as Support Tools for Unit Testing Education Software Engineering Education Zihan Fang Vanderbilt University, Jiliang Li Vanderbilt University, Anda Liang Vanderbilt University, Gina Bai Vanderbilt University, Yu Huang Vanderbilt University | ||
15:20 10mTalk | Creating a Student-Friendly PaaS Platform: Experiences with Tsuru in Software Engineering Education Software Engineering Education Robert Chatley Imperial College London, Jason Bailey Imperial College London, Ivan Procaccini Imperial College London, Zaki Amin Imperial College London, Estibaliz Fraca Imperial College London |
Sirius is located just behind the registration desk.
Facing the registration desk, its entrance is on the right.