Towards an Actionable Flakiness Score
Test flakiness is a prevalent condition within modern build pipelines. To quantify the flakiness, test engineers consider the rate of change of a test’s outcomes. However, this only shows one dimension of the problem: the quantity of test flakiness at a particular point in time. It does not tell us the severity of the flakiness, nor which test outcomes are involved, which is useful information for root cause analysis. In this paper, we propose 3 new encoding-based scores to measure the quantity, severity, and specific outcomes involved in a given instance of test flakiness. Our results show that the performance of the flattened score is comparable to that of pre-existing single-dimensional scores, but analysing them together provides more actionable insights, identifying clusters of tests with similar flakiness behaviour.
Tue 19 MayDisplayed time zone: Seoul change
14:00 - 15:30 | Test FlakinessResearch Papers / Short Papers, Vision and Emerging Results at Room 101 Chair(s): Phil McMinn University of Sheffield | ||
14:00 25mTalk | A Systematic Evaluation of Environmental Flakiness in JavaScript Tests Research Papers Negar Hashemi Massey University, Amjed Tahir Massey University, August Shi The University of Texas at Austin, Shawn Rasheed UCOL, Rachel Blagojevic Massey University | ||
14:25 25mTalk | Pinpointing Flakiness in Web Tests via DOM Event Analysis and LLM-based Reasoning Research Papers Yu Pei University of Luxembourg, Jeongju Sohn Kyungpook National University, Sarra Habchi Cohere, Canada, Mike Papadakis University of Luxembourg | ||
14:50 25mTalk | Automatically Resolving Mismatched Invocations on Test Doubles in Test Suites Research Papers | ||
15:15 15mTalk | Towards an Actionable Flakiness Score Short Papers, Vision and Emerging Results Joanna Kisaakye University of Antwerp, Mutlu Beyazıt University of Antwerp and Flanders Make vzw, Serge Demeyer University of Antwerp and Flanders Make vzw | ||