Reduction of Test Re-runs by Prioritizing Potential Order Dependent Flaky Tests
This program is tentative and subject to change.
Flaky tests can make automated software testing unreliable due to their unpredictable behavior. These tests can pass or fail on multiple runs on the same codebase. However, flaky tests often do not refer to any fault, even though they can make the continuous integration (CI) pipeline fail. A common type of flaky test is the order-dependent (OD) test. The outcome of an OD test depends on the order of its running with respect to other testcases. Several studies have explored the detection and repair of OD tests. However, their methods require multiple re-runs of tests unrelated to the order dependence. Hence, prioritization of potential OD tests is necessary to reduce the re-runs. In this paper, we propose a method to prioritize potential order-dependent tests. By analyzing shared static fields in test classes, we identify tests that are more likely to be order-dependent. In our experiment on 27 projects, our method correctly prioritized all OD tests in 23 projects, reducing the number of test executions by 65.92% on average and cutting re-runs by 72.19%. Our findings show that this approach can make OD test detection more effective in terms of execution cost.
This program is tentative and subject to change.
Sun 27 AprDisplayed time zone: Eastern Time (US & Canada) change
14:00 - 15:30 | |||
14:00 22mPaper | Evaluating NonDex for Modern Java Ecosystem FTW Kaiyao Ke University of Illinois at Urbana-Champaign, Darko Marinov University of Illinois at Urbana-Champaign | ||
14:22 22mTalk | On the Perception and Adoption of Flakiness Scores FTW Joanna Kisaakye , Mutlu Beyazıt University of Antwerp and Flanders Make vzw, Serge Demeyer University of Antwerp and Flanders Make vzw | ||
14:45 22mPaper | Reduction of Test Re-runs by Prioritizing Potential Order Dependent Flaky Tests FTW Hasnain Iqbal University of Dhaka, Zerina Begum University of Dhaka, Kazi Sakib Institute of Information Technology, University of Dhaka | ||
15:07 22mPanel | Mini Panel 2 FTW |