Test Code Flakiness in Mobile Apps: The Developer's Perspective
Context: Test flakiness arises when test cases have a non-deterministic, intermittent behavior that leads them to either pass or fail when run against the same code. While researchers have been contributing to the detection, classification, and removal of flaky tests with several empirical studies and automated techniques, little is known about how the problem of test flakiness arises in mobile applications. Objective: We point out a lack of knowledge on: (1) The prominence and harmfulness of the problem; (2) The most frequent root causes inducing flakiness; and (3) The strategies applied by practitioners to deal with it in practice. An improved understanding of these matters may lead the software engineering research community to assess the need for tailoring existing instruments to the mobile context or for brand-new approaches that focus on the peculiarities identified. Methods: We address this gap of knowledge by means of an empirical study into the mobile developer’s perception of test flakiness. We first perform a systematic grey literature review to elicit how developers discuss and deal with the problem of test flakiness in the wild. Then, we complement the systematic review through a survey study that involves 130 mobile developers and that aims at analyzing their experience on the matter. Results: The results of the grey literature review indicate that developers are often concerned with flakiness connected to user interface elements. In addition, our survey study reveals that flaky tests are perceived as critical by mobile developers, who pointed out major production code- and source code design-related root causes of flakiness, other than the long-term effects of recurrent flaky tests. Furthermore, our study lets the diagnosing and fixing processes currently adopted by developers and their limitations emerge. Conclusion: We conclude by distilling lessons learned, implications, and future research directions.
Thu 30 MayDisplayed time zone: Eastern Time (US & Canada) change
11:00 - 12:40 | Test FlakinessJournal-First Papers / Research Papers / Industry at Room 1 Chair(s): Andrea Stocco Technical University of Munich, fortiss | ||
11:00 20mLong-paper | Test Code Flakiness in Mobile Apps: The Developer's Perspective Journal-First Papers Valeria Pontillo Vrije Universiteit Brussel, Fabio Palomba University of Salerno, Filomena Ferrucci University of Salerno Link to publication | ||
11:20 20mLong-paper | Flakiness goes live: Insights from an In Vivo testing simulation study Journal-First Papers Morena Barboni University of Camerino, Antonia Bertolino National Research Council, Italy, Guglielmo De Angelis CNR-IASI | ||
11:40 20mResearch paper | 262,447 Test Failures Later: An Empirical Evaluation of Flaky Failure Classifiers Research Papers Abdulrahman Alshammari George Mason University, Paul Ammann George Mason University, USA, Michael Hilton Carnegie Mellon University, Jonathan Bell Northeastern University | ||
12:00 20mResearch paper | Automatically Reproducing Timing-Dependent Flaky-Test Failures Research Papers Shanto Rahman The University of Texas at Austin, Aaron Massey George Mason University, Wing Lam George Mason University, August Shi The University of Texas at Austin, Jonathan Bell Northeastern University | ||
12:20 20mIndustry talk | Cost of Flaky Tests in CI: An Industrial Case Study Industry Fabian Leinen Technical University of Munich, Daniel Elsner TU Munich, Alexander Pretschner TU Munich, Andreas Stahlbauer , Michael Sailer , Elmar Juergens CQSE GmbH Pre-print |