Large Language Models for Software Testing Education: an Experience Report
The rapid integration of Large Language Models (LLMs) into software engineering practice is reshaping how software testing activities are performed. LLMs are increasingly used to support software testing. Consequently, software testing education must evolve to prepare students for this new paradigm. However, while students have already begun to use LLMs in an ad hoc manner for testing tasks, there is limited empirical understanding of how such usage influences their testing behaviors, judgment, and learning outcomes. It is necessary to conduct a systematic investigation into how students learn to evaluate, control, and refine LLM-assisted testing results.
This paper presents a mixed-methods, two-phase exploratory study on human–LLM collaboration in software testing education. In Phase I, we analyze classroom learning artifacts and interaction records from 15 students, together with a large-scale survey conducted in a national software testing competition (337 valid responses), to identify recurring prompt-related difficulties across testing tasks. The results reveal systematic interaction breakdowns, including missing contextual information, insufficient constraints, rigid one-shot prompting, and limited strategy-driven iteration, with automated test script generation emerging as a particularly heterogeneous and effort-intensive interaction context. Building on these findings, Phase II conducts an illustrative classroom practice that operationalizes the observed breakdowns into a lightweight, stage-aware prompt scaffold for test script generation, guiding students to explicitly articulate execution-relevant information such as environmental assumptions, interaction grounding, synchronization, and validation intent, and reporting descriptive shifts in students’ testing-related articulation when interacting with LLMs.
This paper empirically characterizes task-dependent difficulties and interaction patterns in students’ use of LLMs for software testing from a learning behavior perspective. It also provides descriptive evidence distinguishing relatively stable interaction contexts from more heterogeneous and unstable ones across testing tasks. Moreover, this paper illustrates a pedagogical direction for translating observed LLM usage difficulties into instructional support, grounded in classroom practice rather than tool-centric optimization. In summary, these findings offer empirical grounding for integrating LLMs into software testing education in a learning-oriented and pedagogically informed manner.
Wed 8 JulDisplayed time zone: Eastern Time (US & Canada) change
14:00 - 15:30 | SEET: LLMs in the SE ClassroomSoftware Engineering Education / Research Papers at MB 9C Chair(s): Felix Dobslaw Chalmers University of Technology | ||
14:00 20mTalk | Large Language Models for Software Testing Education: an Experience Report Software Engineering Education Peng Yang South China Normal University, Yunfeng Zhu Nanjing University, Chao Chang Guangzhou Polytechnic University, Shengcheng Yu Technical University of Munich, Zhenyu Chen Nanjing University, Yong Tang South China Normal University | ||
14:20 20mTalk | An Analysis of Student Perceptions and Learning Impact of Large Language Models in Requirements Engineering Education Software Engineering Education Mohammed Ammar Karimi Dhirubhai Ambani University, formerly DA-IICT Gandhinagar, India, Saurabh Tiwari Dhirubhai Ambani University, formerly DA-IICT Gandhinagar, India, Santosh Singh Rathore ABV-Indian Institute of Information Technology and Management Gwalior | ||
14:40 20mTalk | LLM Use, Cheating, and Academic Integrity in Software Engineering Education Software Engineering Education Ronnie de Souza Santos University of Calgary, Italo Santos University of Hawai‘i at Mānoa, Mariana Pinheiro Bento University of Calgary, Giuseppe Destefanis University College London (UCL), Cleyton Magalhaes Universidade Federal Rural de Pernambuco, Mairieli Wessel Radboud University | ||
15:00 20mTalk | Deliverables Are Not Understanding in the AI Era: Reforming Full-Stack Development Education with Continuous Interactive Learning Software Engineering Education Haolin Jin The University of Sydney, Jiawen Wen The University of Sydney, Zhaoge Bi The University of Sydney, Linghan Hua The University of Sydney, Huaming Chen The University of Sydney | ||