An Empirical Evaluation of Manually Created Equivalent Mutants
Mutation testing consists of evaluating how effective test suites are at detecting artificially seeded defects in the source code, and guiding the improvement of the test suites. Although mutation testing tools are increasingly adopted in practice, equivalent mutants, i.e., mutants that differ only in syntax but not semantics, hamper this process. While prior research investigated how frequent equivalent mutants are produced by mutation testing tools and how effective existing methods of detecting these equivalent mutants are, it remains unclear to what degree humans also create equivalent mutants, and how well they perform at identifying these. We therefore study these questions in the context of Code Defenders, a mutation testing game, in which players competitively produce mutants and tests. Using manual inspection as well as automated identification methods we establish that less than 10% of manually created mutants are equivalent. Surprisingly, our findings indicate that a significant portion of developers struggle to accurately identify equivalent mutants, emphasizing the need for improved detection mechanisms and developer training in mutation testing
Tue 28 MayDisplayed time zone: Eastern Time (US & Canada) change
14:00 - 15:30 | |||
14:00 30mTalk | Test Harness Mutilation Mutation Samuel Moelius Trail of Bits | ||
14:30 30mTalk | An Empirical Evaluation of Manually Created Equivalent Mutants Mutation Philipp Straubinger University of Passau, Alexander Degenhart University of Passau, Gordon Fraser University of Passau Pre-print | ||
15:00 30mTalk | A Study of Flaky Failure De-Duplication to Identify Unreliably Killed Mutants Mutation Abdulrahman Alshammari George Mason University, Paul Ammann George Mason University, USA, Michael Hilton Carnegie Mellon University, Jonathan Bell Northeastern University |