Flaky tests are tests that non-deterministically pass and fail in unchanged code. These tests can be detrimental to developers’ productivity. Particularly when tests run in continuous integration environments, the tests may be competing for access to limited computational resources (CPUs, memory etc.), and we hypothesize that resource (un)-availability may be a significant factor in the failure rate of flaky tests. We present the first assessment of the impact that computational resources have on flaky tests, including a total of 52 projects written in Java, JavaScript and Python, and 27 different resource configurations. Using a rigorous statistical methodology, we determine which tests are RAFTs (Resource-Affected Flaky Tests). We find that 46.5% of the flaky tests in our dataset are RAFTs, indicating that a substantial proportion of flaky-test failures happen depending on the resources available when running tests. We report RAFTs and configurations to avoid them to developers, and received interest to either fix the RAFTs or to improve the specifications of the projects so that tests would be run only in configurations that are unlikely to encounter RAFT failures. Although most test suites in our dataset are executed quite quickly (under one minute) in a baseline configuration, our results highlight the possibility of using this methodology to detect RAFT to reduce the cost of cloud infrastructure for reliably running larger test suites.
| Slides (raft.html) | 3.59MiB |
Wed 25 JunDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
11:00 - 12:30 | Software TestsJournal First / Demonstrations / Research Papers at Pirsenteret 150 Chair(s): Tien N. Nguyen University of Texas at Dallas | ||
11:00 10mTalk | pytest-ranking: A Regression Test Prioritization Tool for Python Demonstrations Runxiang Cheng University of Illinois at Urbana-Champaign, Kaiyao Ke University of Illinois Urbana-Champaign, Darko Marinov University of Illinois at Urbana-Champaign | ||
11:10 20mTalk | The Effects of Computational Resources on Flaky Tests Journal First Denini Silva Federal University of Pernambuco, Martin Gruber BMW Group, Satyajit Gokhale Northeastern University, Ellen Arteca Northeastern University, Alexi Turcotte CISPA, Marcelo d'Amorim North Carolina State University, Wing Lam George Mason University, Stefan Winter LMU Munich, Jonathan Bell Northeastern University DOI File Attached | ||
11:30 20mTalk | Prioritizing Speech Test Cases Journal First Zhou Yang Singapore Management University; University of Alberta, Jieke Shi Singapore Management University, Muhammad Hilmi Asyrofi School of Computing and Information Systems, Singapore Management University, Bowen Xu North Carolina State University, Xin Zhou Singapore Management University, Singapore, DongGyun Han Royal Holloway, University of London, David Lo Singapore Management University | ||
11:50 20mTalk | Automated Unit Test Refactoring Research Papers Yi Gao Zhejiang University, Xing Hu Zhejiang University, Xiaohu Yang Zhejiang University, Xin Xia Zhejiang University DOI | ||
12:10 20mTalk | Understanding and Characterizing Mock Assertions in Unit Tests Research Papers Hengcheng Zhu The Hong Kong University of Science and Technology, Valerio Terragni University of Auckland, Lili Wei McGill University, Shing-Chi Cheung Hong Kong University of Science and Technology, Jiarong Wu , Yepang Liu Southern University of Science and Technology DOI Pre-print | ||
This room is located outside Clarion Hotel
This room is located in the Pirsenteret (The Pier Center) convention center. It is just outside the hotel, on the back, towards the fjord.
You should be able to go through the emergency exit at Clarion, just on the side of the Cosmos 3 wing, which will be bring you close to Pirsenteret.
The entrance to the center is from here:
https://maps.app.goo.gl/dU3qH6kAimXGBNHe7
Once inside, go all straight and you will find signage to reach the room. The room is known as room 150 inside the center.