Write a Blog >>
ICSE 2021
Mon 17 May - Sat 5 June 2021
Thu 27 May 2021 15:05 - 15:25 at Blended Sessions Room 4 - 3.3.4. Testing: Flaky Tests Chair(s): José Miguel Rojas
Fri 28 May 2021 03:05 - 03:25 at Blended Sessions Room 4 - 3.3.4. Testing: Flaky Tests

Code coverage measures the degree to which source code elements (e.g., statements, branches) are invoked during testing. Despite growing evidence that coverage is a problematic measurement, it is often used to make decisions about where testing effort should be invested. For example, using coverage as a guide, tests should be written to invoke the non-covered program elements. At their core, coverage measurements assume that invocation of a program element during any test is equally valuable. Yet in reality, some tests are more robust than others. As an instance of this, we posit that program elements that are only covered by flaky tests, i.e., tests with non-deterministic behaviour, are also worthy of investment of additional testing effort.

We set out to quantify, characterize, and mitigate “flakily covered” program elements (i.e., those elements that are only covered by flaky tests). To that end, we perform an empirical study of three large software systems from the OpenStack community. In terms of quantification, we find that systems are disproportionately impacted by flakily covered statements with 5% and 10% of the covered statements in Nova and Neutron being flakily covered, respectively, while <1% of Cinder statements are flakily covered. In terms of characterization, we find that incidences of flakily covered statements could not be well explained by solely using code characteristics, such as dispersion, ownership, and development activity. In terms of mitigation, we propose GreedyFlake – a test effort prioritization algorithm to maximize return on investment when tackling the problem of flakily covered program elements. We find that GreedyFlake outperforms baseline approaches by at least eight percentage points of Area Under the Cost Effectiveness Curve.

Conference Day
Thu 27 May

Displayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change

15:05 - 16:05
3.3.4. Testing: Flaky TestsTechnical Track / Journal-First Papers at Blended Sessions Room 4 +12h
Chair(s): José Miguel RojasUniversity of Leicester, UK
15:05
20m
Paper
Quantifying, Characterizing, and Mitigating Flakily Covered Program ElementsJournal-First
Journal-First Papers
Pre-print Media Attached
15:25
20m
Paper
FlakeFlagger: Predicting Flakiness Without Rerunning TestsArtifact ReusableTechnical TrackArtifact Available
Technical Track
Abdulrahman AlshammariGeorge Mason University, Christopher MorrisCarnegie Mellon University, Michael HiltonCarnegie Mellon University, USA, Jonathan BellNortheastern University
Pre-print Media Attached
15:45
20m
Paper
An Empirical Analysis of UI-based Flaky TestsArtifact ReusableTechnical TrackArtifact Available
Technical Track
Alan RomanoUniversity at Buffalo, Zihe SongUniversity of Texas at Dallas, Sampath GrandhiUniversity of Texas at Dallas, Wei YangUniversity of Texas at Dallas, Weihang WangUniversity at Buffalo, SUNY
Pre-print Media Attached

Conference Day
Fri 28 May

Displayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change

03:05 - 04:05
03:05
20m
Paper
Quantifying, Characterizing, and Mitigating Flakily Covered Program ElementsJournal-First
Journal-First Papers
Pre-print Media Attached
03:25
20m
Paper
FlakeFlagger: Predicting Flakiness Without Rerunning TestsArtifact ReusableTechnical TrackArtifact Available
Technical Track
Abdulrahman AlshammariGeorge Mason University, Christopher MorrisCarnegie Mellon University, Michael HiltonCarnegie Mellon University, USA, Jonathan BellNortheastern University
Pre-print Media Attached
03:45
20m
Paper
An Empirical Analysis of UI-based Flaky TestsArtifact ReusableTechnical TrackArtifact Available
Technical Track
Alan RomanoUniversity at Buffalo, Zihe SongUniversity of Texas at Dallas, Sampath GrandhiUniversity of Texas at Dallas, Wei YangUniversity of Texas at Dallas, Weihang WangUniversity at Buffalo, SUNY
Pre-print Media Attached