Mutation 2025

The concept of mutation testing was first proposed over 40 years ago. There exists theoretical evidence that the technique is superior to structural coverage criteria as a means of assessing and improving the quality of an existing test suite. However, due to factors such as computational cost and the lack of availability of mature tools for many programming languages, mutation technique has not caught on as a standard practice in industry. In recent years, there has been significant progress with regard to cost and tool availability, potentially making the adoption of mutation testing more realistic in practice. Rather, we now hypothesize that such adoption is hindered by a lack of guidance on how to effectively and efficiently utilize mutation testing in a development and testing workflow.

In this talk, I will discuss the results of a two year collaboration with Zenseact - a developer of autonomous driving and advanced driver assistance systems - to implement mutation testing in their continuous integration pipeline and to explore how mutation testing can be integrated into the testing process. This collaboration has illustrated the potential of mutation testing in industry, and yielded a number of findings related to the technical integration of mutation testing, how and when mutation testing should be applied, how the results of mutation testing should be presented, and how the results should be applied. However, many research challenges remain to be solved before mutation testing will be used in "standard" practice.

Gregory Gay is an Associate Professor in the Interaction Design and Software Engineering division in the Department of Computer Science and Engineering at Chalmers University of Technology and the University of Gothenburg. His research interests include software testing and analysis, AI and search-based automation of development tasks, and AI engineering - all with the aim of helping developers deliver complex systems in a safe, secure, and efficient manner. His research has been funded by agencies including the National Science Foundation (USA), the Swedish Research Council, and the Wallenberg AI, Autonomous Systems, and Software Program (WASP). In addition, he has an extensive history of industrial research collaboration, with partners including Ericsson, Zenseact, Volvo Cars, Rockwell Collins, and NASA, among others. His current service roles include deputy editor-in-chief of Automated Software Engineering Journal, the co-chair of the Software Technology research cluster in WASP, and the steering committee of the International Conference on Software Testing (ICST). In the past, he has also served as the program co-chair for ICST and the Symposium on Search-Based Software Engineering (SSBSE), on the SSBSE steering committee, and in a number of other roles in the software engineering research community.

This talk will cover Meta’s work on the Automated Compliance Hardening (ACH) tool, which uses mutation testing to guide Assured LLM-based Software Engineering. ACH generates relatively few mutants (aka simulated faults), compared to traditional mutation testing. Instead, it focuses on generating currently undetected faults that are specific to an issue of concern. From these currently uncaught faults, ACH generates tests that can catch them, thereby `killing' the mutants and consequently hardening the platform against regressions. ACH also deploys an LLM-based equivalent mutant detection agent that achieves a precision of 0.79 and a recall of 0.47 (rising to 0.95 and 0.96 with simple pre-processing). ACH was used by Messenger and WhatsApp test-a-thons where engineers accepted 73% of its tests, judging 36% to relevant. The talk will review Assured LLMSE, LLM-based test generation and mutation testing work at Meta.

Mark Harman is a Research Scientist at Meta London and a professor at University College London. He joined Meta following acquisition of his startup Majicke. He has published over 300 papers, with over 45,000 citations, and an H index of 105, making him the most highly cited scientist in the field of both Software Testing and of Program Analysis. His work has been deployed throughout Meta’s platforms for the past eight years, directly impacting over 3 billion people who rely on its product’s for social networking, community building and communication. His work has also directly impacted more than 200 million small companies that use Instagram, Facebook and WhatsApp to reach their customers and indirectly impacted many others that have deployed technology based on it, such as Microsoft, Google and Amazon. For his scientific work, Harman received the IEEE Harlan Mills Award and the ACM Outstanding Research Award in 2019. In 2020, he was elected a fellow of the Royal Academy of Engineering.