Deep Learning (DL) solutions are increasingly adopted, but how
to test them remains a major open research problem. Existing and
new testing techniques have been proposed for and adapted to DL
systems, including mutation testing. However, no approach has
investigated the possibility to simulate the effects of real DL faults
by means of mutation operators.
We have defined 35 DL mutation operators relying on 3 empirical
studies about real faults in DL systems. We followed a systematic
process to extract the mutation operators from the existing fault
taxonomies, with a formal phase of conflict resolution in case of
disagreement. We have implemented 24 of these DL mutation operators
into DeepCrime, the first source-level pre-training mutation
tool based on real DL faults. We have assessed our mutation operators
to understand their characteristics: whether they produce
interesting, i.e., killable but not trivial, mutations. Then, we have
compared the sensitivity of our tool to the changes in the quality
of test data with that of DeepMutation++, an existing post-training
DL mutation tool.