Mutation Testing in Continuous Integration: An Exploratory Industrial Case Study
Despite its potential quality benefits, the cost of mutation testing and the immaturity of mutation tools for many languages have led to a lack of adoption in industrial software development. In an exploratory case study at Zenseact—a company in the automotive domain—we have explored how mutation testing could be effectively applied in a typical Continuous Integration-based workflow. We evaluated the capabilities of C++ mutation tools, and demonstrate their use in GitHub Actions-based CI workflows. Our investigation reveals that Dextool and Mull could be used in a CI workflow. Additionally, we conducted an interview study to understand how developers would use mutation testing in their CI workflows. Based on our qualitative analysis and practices proposed in literature, we discuss recommendations to integrate mutation testing in a CI workflow. For instance, visualising trends in the mutation score enable practitioners to understand how test quality is evolving. Moreover, tools should have a balance between offering fast feedback and keeping or flagging relevant mutants. Lastly, practitioners raised the need that the mutation should be applied at commit level, and that developers inexperienced with mutation testing should be trained in the implications of the practice.
Sun 16 AprDisplayed time zone: Dublin change
14:00 - 15:30 | |||
14:00 30mTalk | Mutation Testing in Continuous Integration: An Exploratory Industrial Case Study Mutation Jonatan Örgård Chalmers | University of Gothenburg, Gregory Gay Chalmers | University of Gothenburg, Francisco Gomes de Oliveira Neto Chalmers University of Technology, Sweden / University of Gothenburg, Sweden, Kim Viggedal Zenseact | ||
14:30 30mTalk | Validation of Mutation Testing in the Safety Critical Industry through a Pilot Study Mutation Sten Vercammen University of Antwerp, Belgium, Markus Borg CodeScene, Serge Demeyer University of Antwerp; Flanders Make | ||
15:00 30mAwards | Best paper award and closing Mutation |