ISSTA 2022
Mon 18 - Fri 22 July 2022 Online

Regression testing is an important part of the software development process but suffers from the presence of flaky tests. Flaky tests are tests that can nondeterministically pass or fail regardless of code changes. Order-dependent flaky tests are a prominent kind of flaky tests whose outcome depends on the test order in which they are run. Prior work has focused on detecting order-dependent flaky tests through rerunning all tests in different test orders on a single version of code. As code is constantly changing, rerunning all tests in different test orders after every change is costly.

In this work, we propose IncIDFlakies, a technique to detect order-dependent flaky tests by analyzing code changes to detect newly-introduced order-dependent flaky tests due to those changes. Building upon existing work in iDFlakies that reruns tests in different test orders, IncIDFlakies analyzes and selects to run only the tests that (1) are affected by the change, and (2) can potentially result in a test-order dependency among each other due to potential shared state. Running IncIDFlakies on 67 order-dependent flaky tests across changes in code in their respective projects, including the changes where they became flaky, we find that IncIDFlakies can select to run on average 65.4% of all the tests, resulting in running 68.4% of the time that baseline iDFlakies would use when running the same number of test orders with the full test suite. Furthermore, we find that IncIDFlakies can still ensure that the test orders it runs can potentially detect the order-dependent flaky tests.

Wed 20 Jul

Displayed time zone: Seoul change

01:20 - 02:40
Session 1-1: Oracles, Models, and Measurement DTechnical Papers at ISSTA 1
01:20
20m
Talk
Combining Solution Reuse and Bound Tightening for Efficient Analysis of Evolving SystemsACM SIGSOFT Distinguished Paper
Technical Papers
Clay Stevens University of Nebraska-Lincoln, Hamid Bagheri University of Nebraska-Lincoln
DOI
01:40
20m
Talk
Evolution-Aware Detection of Order-Dependent Flaky Tests
Technical Papers
Chengpeng Li University of Texas at Austin, August Shi University of Texas at Austin
DOI
02:00
20m
Talk
jTrans: Jump-Aware Transformer for Binary Code Similarity Detection
Technical Papers
Hao Wang Tsinghua University, Wenjie Qu Huazhong University of Science and Technology, Gilad Katz Ben-Gurion University of the Negev, Wenyu Zhu Tsinghua University, Zeyu Gao University of Science and Technology of China, Han Qiu Tsinghua University, Jianwei Zhuge Tsinghua University, Chao Zhang Tsinghua University
DOI Pre-print
02:20
20m
Talk
On the Use of Evaluation Measures for Defect Prediction Studies
Technical Papers
Rebecca Moussa University College London, Federica Sarro University College London
DOI
07:00 - 08:20
Session 2-1: Oracles, Models, and Measurement ETechnical Papers at ISSTA 1
Chair(s): Christoph Csallner University of Texas at Arlington
07:00
20m
Talk
On the Use of Evaluation Measures for Defect Prediction Studies
Technical Papers
Rebecca Moussa University College London, Federica Sarro University College London
DOI
07:20
20m
Talk
Combining Solution Reuse and Bound Tightening for Efficient Analysis of Evolving SystemsACM SIGSOFT Distinguished Paper
Technical Papers
Clay Stevens University of Nebraska-Lincoln, Hamid Bagheri University of Nebraska-Lincoln
DOI
07:40
20m
Talk
Evolution-Aware Detection of Order-Dependent Flaky Tests
Technical Papers
Chengpeng Li University of Texas at Austin, August Shi University of Texas at Austin
DOI
08:00
20m
Talk
FDG: A Precise Measurement of Fault Diagnosability Gain of Test Cases
Technical Papers
Gabin An KAIST, Shin Yoo KAIST
DOI Pre-print