Large Language Model (LLM) pipelines can generate semantically meaningful unit tests with high coverage, but generated tests often fail to compile or execute. Many frameworks therefore use LLM-driven repair loops that iteratively re-prompt on failures until validation succeeds, but these repairs can introduce issues such as hallucinating an invented class solely to make the test pass. In this paper, we study repair-induced issues and effects of prompt strategy across three Java projects. We build a dataset of LLM-generated broken-to-fixed test pairs, where initially failing tests are repaired via a loop under zero-shot, few-shot, and chain-of-thought prompting, derive a taxonomy from 117 manually analyzed pairs (three anti-pattern categories, seven scenarios), and compare the performance of prompting strategies through repair rate and post-repair coverage.
Program Display Configuration
Wed 8 Jul
Displayed time zone: Eastern Time (US & Canada)change