Automated Test Suite Enhancement Using Large Language Models with Few-shot Prompting
Unit testing is essential for verifying the functional correctness of code modules (e.g., classes, methods), but manual test development is often labour-intensive and time-consuming. Unit tests generated by tools that employ traditional approaches, such as search-based software testing (SBST), lack readability, naturalness, and practical usability. LLMs have recently provided promising results and become integral to developers’ daily practices. Consequently, software repositories now include a mix of human-written tests, LLM-generated tests, and those from tools employing traditional approaches such as SBST. While LLMs’ zero-shot capabilities have been widely studied, their few-shot learning potential for unit test generation remains underexplored. Few-shot prompting enables LLMs to learn from examples in the prompt, and automatically retrieving such examples could enhance test suites. This study investigates how the origin of few-shot examples—human, SBST, or LLM—affects the quality of generated tests. We conducted experiments on HumanEval and ClassEval datasets using GPT-4.o, which is integrated into GitHub Copilot and widely used among developers. During our experiments, we evaluated the functional correctness, coverage, and quality of generated tests and their contribution to improving existing test suites. We also assessed retrieval-based methods for selecting relevant examples. Our results show that LLMs can generate high-quality tests via few-shot prompting, with human-written examples producing the best coverage and correctness. Additionally, selecting examples based on the combined similarity of problem description and code consistently yields the most effective few-shot prompts.
| Slides (ICPC2026_chudic-and-calikli.pdf) | 6.28MiB |