ISSTA 2019
Mon 15 - Fri 19 July 2019 Beijing, China
Thu 18 Jul 2019 11:22 - 11:45 at Grand Ballroom - Regression Testing Chair(s): Dan Hao

Mutation testing is widely used in research as a metric for evaluating the quality of test suites. However, traditional mutation testing assumes tests to exhibit deterministic behavior, in terms of their coverage and the outcome of a test (not) killing a certain mutant. Such an assumption does not hold in the presence of flaky tests, whose outcomes can non-deterministically differ even when run on the same code under test. Almost all modern software projects have some flaky tests. Without reliable test outcomes, mutation testing can result in unreliable results, e.g., in our experiments, mutation scores vary by 5 percentage points on average between repeated executions, and the difference in mutant-test pairs was 10 percentage points on average. We propose an advanced technique that better controls for flakiness throughout the mutation testing process. We implement our techniques by modifying the popular open-source tool, PIT. We evaluate our modifications on 30 open-source projects, finding that our technique can increase developers’ confidence in mutation results in the presence of flaky tests by almost entirely eliminating the number of “unknown” (flaky) mutants.

Thu 18 Jul
Times are displayed in time zone: Beijing, Chongqing, Hong Kong, Urumqi change

11:00 - 12:30: Regression TestingTechnical Papers at Grand Ballroom
Chair(s): Dan HaoPeking University
11:00 - 11:22
Root Causing Flaky Tests in a Large-scale Industrial Setting
Technical Papers
Wing LamUniversity of Illinois at Urbana-Champaign, Patrice GodefroidMicrosoft Research, Suman NathMicrosoft Corporation, Anirudh SanthiarIndian Institute of Science, Suresh Thummalapenta
11:22 - 11:45
Mitigating the Effects of Flaky Tests on Mutation Testing
Technical Papers
August ShiUniversity of Illinois at Urbana-Champaign, Jonathan BellGeorge Mason University, Darko MarinovUniversity of Illinois at Urbana-Champaign
Pre-print Media Attached
11:45 - 12:07
Assessing the State and Improving the Art of Parallel Testing for CArtifacts ReusableArtifacts Functional
Technical Papers
Oliver SchwahnTU Darmstadt, Nicolas CoppikTU Darmstadt, Stefan WinterTU Darmstadt, Neeraj Suri
12:07 - 12:30
Failure Clustering Without Coverage
Technical Papers