Recent advances in defect detection use language models. Here, existing works enhanced the training data to improve the models’ robustness when applied to semantically identical code (i.e., the prediction should be the same). However, the use of semantically identical code has not been considered for improving the tools during their application - a concept closely related to metamorphic testing.

The goal of our study is to determine whether we can use semantic-preserving transformations, analogue to mutation operators, to improve the performance of defect detection tools in the testing stage. We first collect existing publications which im- plemented semantic-preserving transformations and share their implementation, such that we can reuse them. We empirically study the effectiveness of three different ensemble strategies for enhancing defect detection tools. We apply the collected transfor- mations on a popular dataset (Devign), considering vulnerabilities as a type of defect, and use two fine-tuned large language models for defect detection (VulBERTa, PLBART).

We found 28 publications with 94 different transformation. We choose to implement 39 transformations, but a manual check revealed that 23 out 39 transformations change code semantics. Using the 16 remaining, correct transformations and three ensemble strategies, we were not able to increase the accuracy of the defect detection models. Our results show that reusing shared semantic-preserving transformation is difficult, sometimes even causing wrongful changes to the semantics.