ICST 2024
Mon 27 - Fri 31 May 2024 Canada
Thu 30 May 2024 11:00 - 11:20 at Room 1 - Test Flakiness Chair(s): Andrea Stocco

Context: Test flakiness arises when test cases have a non-deterministic, intermittent behavior that leads them to either pass or fail when run against the same code. While researchers have been contributing to the detection, classification, and removal of flaky tests with several empirical studies and automated techniques, little is known about how the problem of test flakiness arises in mobile applications. Objective: We point out a lack of knowledge on: (1) The prominence and harmfulness of the problem; (2) The most frequent root causes inducing flakiness; and (3) The strategies applied by practitioners to deal with it in practice. An improved understanding of these matters may lead the software engineering research community to assess the need for tailoring existing instruments to the mobile context or for brand-new approaches that focus on the peculiarities identified. Methods: We address this gap of knowledge by means of an empirical study into the mobile developer’s perception of test flakiness. We first perform a systematic grey literature review to elicit how developers discuss and deal with the problem of test flakiness in the wild. Then, we complement the systematic review through a survey study that involves 130 mobile developers and that aims at analyzing their experience on the matter. Results: The results of the grey literature review indicate that developers are often concerned with flakiness connected to user interface elements. In addition, our survey study reveals that flaky tests are perceived as critical by mobile developers, who pointed out major production code- and source code design-related root causes of flakiness, other than the long-term effects of recurrent flaky tests. Furthermore, our study lets the diagnosing and fixing processes currently adopted by developers and their limitations emerge. Conclusion: We conclude by distilling lessons learned, implications, and future research directions.

Thu 30 May

Displayed time zone: Eastern Time (US & Canada) change

11:00 - 12:40
Test FlakinessJournal-First Papers / Research Papers / Industry at Room 1
Chair(s): Andrea Stocco Technical University of Munich, fortiss
11:00
20m
Long-paper
Test Code Flakiness in Mobile Apps: The Developer's Perspective
Journal-First Papers
Valeria Pontillo Vrije Universiteit Brussel, Fabio Palomba University of Salerno, Filomena Ferrucci University of Salerno
Link to publication
11:20
20m
Long-paper
Flakiness goes live: Insights from an In Vivo testing simulation study
Journal-First Papers
Morena Barboni University of Camerino, Antonia Bertolino National Research Council, Italy, Guglielmo De Angelis CNR-IASI
11:40
20m
Research paper
262,447 Test Failures Later: An Empirical Evaluation of Flaky Failure Classifiers
Research Papers
Abdulrahman Alshammari George Mason University, Paul Ammann George Mason University, USA, Michael Hilton Carnegie Mellon University, Jonathan Bell Northeastern University
12:00
20m
Research paper
Automatically Reproducing Timing-Dependent Flaky-Test Failures
Research Papers
Shanto Rahman The University of Texas at Austin, Aaron Massey George Mason University, Wing Lam George Mason University, August Shi The University of Texas at Austin, Jonathan Bell Northeastern University
12:20
20m
Industry talk
Cost of Flaky Tests in CI: An Industrial Case Study
Industry
Fabian Leinen Technical University of Munich, Daniel Elsner TU Munich, Alexander Pretschner TU Munich, Andreas Stahlbauer , Michael Sailer , Elmar Juergens CQSE GmbH
Pre-print