ICST 2024
Mon 27 - Fri 31 May 2024 Canada
Tue 28 May 2024 11:25 - 11:50 at Room 5 - CCIW Session 2 Chair(s): Tim A. D. Henderson

Flaky tests are tests that nondeterministically pass and fail in unchanged code. These tests can be detrimental to developers’ productivity. Particularly when tests run in continuous integration environments, the tests may be competing for access to limited computational resources (CPUs, memory etc.), and we hypothesize that resource (in)availability may be a significant factor in the failure rate of flaky tests. We present the first assessment of the impact that computational resources have on flaky tests, including a total of 52 projects written in Java, JavaScript and Python, and 27 different resource configurations. Using a rigorous statistical methodology, we determine which tests are RAFTs (Resource-Affected Flaky Tests). We find that 46.5% of the flaky tests in our dataset are RAFTs, indicating that a substantial proportion of flaky-test failures can be avoided by adjusting the resources available when running tests. We report RAFTs and configurations to avoid them to developers, and received interest to either fix the RAFTs or to improve the specifications of the projects so that tests would be run only in configurations that are unlikely to encounter RAFT failures. Our results also have implications for researchers attempting to detect flaky tests, e.g., reducing the resources available when running tests is a cost-effective approach to detect more flaky failures.

Slides (ICST CCIW 2024 RAFT.pdf)3.26MiB

Tue 28 May

Displayed time zone: Eastern Time (US & Canada) change

11:00 - 12:30
CCIW Session 2CCIW at Room 5
Chair(s): Tim A. D. Henderson Google
11:00
25m
Talk
Future of Google Testing
CCIW
11:25
25m
Talk
The Effects of Computational Resources on Flaky Tests
CCIW
Jonathan Bell Northeastern University, Wing Lam George Mason University, Marcelo d'Amorim North Carolina State University, Alexi Turcotte Northeastern University, Ellen Arteca Google MTV , Satyajit Gokhale Northeastern University, Martin Gruber BMW Group, University of Passau, Denini Silva Federal University of Pernambuco
Media Attached File Attached
11:50
40m
Panel
What are the biggest challenges in large scale CI/CD?
CCIW