AST 2023
Mon 15 - Tue 16 May 2023 Melbourne, Australia
co-located with ICSE 2023
Tue 16 May 2023 11:22 - 11:45 at Meeting Room 107 - Test Flakiness

Non-deterministically behaving (i.e., flaky) tests hamper regression testing as they destroy trust and waste computational and human resources. Eradicating flakiness in test suites is therefore an important goal, but automated debugging tools are needed to support developers when trying to understand the causes of flakiness. A popular example for an automated approach to support regular debugging is spectrum-based fault localization (SFL), a technique that identifies software components that are most likely the causes of failures. While it is possible to also apply SFL for locating likely sources of flakiness in code, unfortunately the flakiness makes SFL both imprecise and nondeterministic. In this paper we introduce SFFL (Spectrum-based Flaky Fault Localization), an extension of traditional coveragebased SFL that exploits our observation that 80% of flaky tests exhibit varying coverage behavior between different runs. By distinguishing between stable and flaky coverage, SFFL is able to locate the sources of flakiness more precisely and keeps the localization itself deterministic. An evaluation on 101 flaky tests taken from 48 open-source Python projects demonstrates that SFFL is effective: Of five prominent SFL formulas, DStar, Ochiai, and Op2 yield the best overall performance. On average, they are able to narrow down the fault’s location to 3.5 % of the project’s code base, which is 18.7 % better than traditional SFL (for DStar). SFFL’s effectiveness depends however on the root causes of flakiness: The source of non-order-dependent flaky tests can be located far more precisely than order-dependent faults.

Tue 16 May

Displayed time zone: Hobart change

11:00 - 12:30
Test FlakinessAST 2023 at Meeting Room 107
11:00
22m
Talk
On the Effect of Instrumentation on Test Flakiness
AST 2023
Shawn Rasheed Universal College of Learning, Jens Dietrich Victoria University of Wellington, Amjed Tahir Massey University
Pre-print
11:22
22m
Talk
Debugging Flaky Tests using Spectrum-based Fault Localization
AST 2023
Martin Gruber BMW Group, University of Passau, Gordon Fraser University of Passau
Pre-print
11:45
22m
Talk
FlakyCat: Predicting Flaky Tests Categories using Few-Shot Learning
AST 2023
Amal Akli University of Luxembourg, Guillaume Haben University of Luxembourg, Sarra Habchi Ubisoft, Mike Papadakis University of Luxembourg, Luxembourg, Yves Le Traon University of Luxembourg, Luxembourg
12:07
22m
Talk
Detecting Potential User-data Save & Export Losses due to Android App Termination
AST 2023
Sydur Rahaman New Jersey Institute of Technology, Umar Farooq University of California at Riverside, Iulian Neamtiu New Jersey Institute of Technology, Zhijia Zhao University of California at Riverside