Improving Automated Patch Correctness Assessment by Designing LLM-Based Oracles

Automated Program Repair (APR) suffers from the overfitting problem, in which generated patches pass the available test suite while remaining semantically incorrect. FixCheck, a recent Automated Patch Correctness Assessment (APCA) technique, aims to mitigate this issue by leveraging large language models (LLMs) to generate additional tests that expose incorrect patches. However, our empirical analysis shows that FixCheck’s LLM-based assertion generation frequently produces semantically weak assertions or non-compiling tests, substantially limiting its effectiveness. To address these limitations, we propose six prompt engineering strategies and systematically evaluate them on 109 incorrect patches from the Defects4J benchmark using LLMs with varying capacities, including GPT-4o, GPT-4o-mini, and Llama~3.2~3B. Our results demonstrate that the effectiveness of prompt engineering is strongly model-dependent. For GPT-based models, explicitly aligning the APCA objective through role definition improves incorrect patch detection by up to 7.5%. In contrast, for Llama~3.2~3B, enforcing strict output format constraints with illustrative examples reduces non-compiling assertions and improves detection performance by up to 19.9%. Overall, this study provides practical, model-aware prompt design guidelines for building reliable LLM-based APCA systems.