Mutation Testing is primarily motivated by its ability to simulate mistakes commonly made by experienced programmers. The potential number of such mistakes in any real-world program is enormous, making it impractical to simulate all of them. Therefore, only a subset of faults close to the correct program is imitated using mutation operators. This theory is grounded in two principles: i) Coupling Effect: If a test suite detects all simple faults, it will likely detect most complex faults; and ii) Competent Programmer Hypothesis: Most programmers are competent enough to create correct or nearly correct source code. This implies that faults should usually manifest through small variations of the correct code. Consequently, researchers assumed that the synthetic faults injected in source code through the mutation operators closely resemble the real faults. Unfortunately, it is still unclear whether the competent programmer hypothesis holds, as past research presents contradictory claims.

In this study, we provide a new perspective on the competent programmer hypothesis and its relation to mutation testing. We try to re-create real-world faults through chains of mutations to understand if there is a direct link between mutation testing and faults. The lengths of these paths help us to understand if the source code is really almost correct, or if large variations are required. Our experiments used a state-of-the-art benchmark database of real faults named Defects4J 2.0.0. It contains 835 reproducible real-world faults in 17 open-source projects that comprise a total of 1044 bug-fix pairs of files. By examining the length of mutation chains, we aim to gain insights into the correlation between mutation testing and actual fault behavior.

We observe that 7% of real faults are fully recreated, 71% are partially recreated, while the remaining 22% are not recreated. On the one hand, this problem can be generalized as the key reason for failed reproductions: whenever there is a change to an external dependency that is outside of the scope of a method that is modified as part of a bug fix, mutation testing is likely not able to reproduce such faults. On the other hand, we found when we could fully reproduce faults, the mutation paths were relatively short. The partial reproductions also contained longer mutation paths, and mostly contained between 5% and 75% of the required mutations. Our results indicate that while the competent programmer hypothesis seems to be true, mutation testing is missing important mutation operators to generate representative real-world faults.