Ripples of a Mutation — An Empirical Study of Propagation Effects in Mutation Testing
The mechanics of how a fault reveals itself as a test failure is of keen interest to software researchers and practitioners alike. An improved understanding of how faults translate to failures can guide improvements in broad facets of software testing, ranging from test suite design to automated program repair, which are premised on the understanding that the presence of faults would alter some test executions. In this work, we study such effects by mutations, as applicable in mutation testing. Mutation testing enables the generation of a large corpus of faults; thereby harvesting a large pool of mutated test runs for analysis. Specifically, we analyze more than 1.1 million mutated test runs to study if and how the underlying mutations induce infections that propagate their way to observable failures. We adopt a broad-spectrum approach to analyze such a large pool of mutated runs. For every mutated test run, we are able to determine: (a) if the mutation induced a state infection; (b) if the infection propagated through the end of the test run; and (c) if the test failed in the presence of a propagated infection. By examining such infection-, propagation- and revealability- effects for more than 43,000 mutations executed across 1.1 million test runs, we are able to arrive at some surprising findings. Our results find that once state infection is observed, propagation is frequently detected; however, a propagated infection does not always reveal itself as a test failure. We also find that a significant portion of survived mutants in our study could have been killed by observing propagated state infections that were left undetected. Finally, we also find that different mutation operators can demonstrate substantial differences in their specific impacts on the execution-to-failure ripples of the resulting mutations.
Wed 17 AprDisplayed time zone: Lisbon change
14:00 - 15:30 | Testing 2Research Track / Software Engineering Education and Training / Software Engineering in Practice / Demonstrations / Journal-first Papers at Eugénio de Andrade Chair(s): Jonathan Bell Northeastern University | ||
14:00 15mTalk | Ripples of a Mutation — An Empirical Study of Propagation Effects in Mutation Testing Research Track Hang Du University of California at Irvine, Vijay Krishna Palepu Microsoft, James Jones University of California at Irvine DOI | ||
14:15 15mTalk | Fast Deterministic Black-box Context-free Grammar Inference Research Track Mohammad Rifat Arefin The University of Texas at Arlington, Suraj Shetiya University of Texas at Arlington, Zili Wang Iowa State University, Christoph Csallner University of Texas at Arlington Pre-print Media Attached | ||
14:30 15mTalk | Bridging Theory to Practice in Software Testing Teaching through Team-based Learning (TBL) and Open Source Software (OSS) Contribution Software Engineering Education and Training | ||
14:45 15mTalk | Productive Coverage: Improving the Actionability of Code Coverage Software Engineering in Practice Marko Ivanković Google; Universität Passau, Goran Petrović Google Inc, Yana Kulizhskaya Google Inc, Mateusz Lewko Google Inc, Luka Kalinovčić No affiliation, René Just University of Washington, Gordon Fraser University of Passau | ||
15:00 15mTalk | Taming Timeout Flakiness: An Empirical Study of SAP HANA Software Engineering in Practice Pre-print | ||
15:15 7mTalk | Testing Abstractions for Cyber-Physical Control Systems Journal-first Papers Claudio Mandrioli University of Luxembourg, Max Nyberg Carlsson Lund University, Martina Maggio Saarland University, Germany / Lund University, Sweden Pre-print | ||
15:22 7mTalk | FaultFuzz: A Coverage Guided Fault Injection Tool for Distributed Systems Demonstrations Wenhan Feng Institute of Software, Chinese Academy of Sciences, Qiugen Pei Joint Laboratory on Cyberspace Security China Southern Power Grid, Yu Gao Institute of Software, Chinese Academy of Sciences, China, Dong Wang Institute of software, Chinese academy of sciences, Wensheng Dou Institute of Software Chinese Academy of Sciences, Jun Wei Institute of Software at Chinese Academy of Sciences; University of Chinese Academy of Sciences; University of Chinese Academy of Sciences Chongqing School, Zheheng Liang Joint Laboratory on Cyberspace Security of China Southern Power Grid, Zhenyue Long Joint Laboratory on Cyberspace Security China Southern Power Grid Pre-print |