Background: Test flakiness is a major problem in the software industry. Flaky tests fail seemingly at random without changes to the code and thus impede continuous integration (CI). Some researchers argue that all tests can be considered flaky and that tests only differ in their frequency of flaky failures. This position implies that the definition of test flakiness includes failures caused by interruptions in the testing environment.

Aims: With the goal of developing mitigation strategies to reduce the negative impact of test flakiness, we study characteristics of tests and the test environment which potentially impact test flakiness.

Method: We construct two datasets based on SAP HANA’s test results over a 12-week period: one based on production data of the SAP HANA CI pipeline, the other based on targeted test executions from a dedicated flakiness experiment. We conduct correlation analysis for test and test environment characteristics with respect to their influence on the frequency of flaky test failures.

Results: In our study, the average test execution time had the strongest positive correlation with the test flakiness rate (r = 0.79), which confirms previous studies. Potential reasons for higher flakiness include the larger test scope of long-running tests or test executions on a slower test infrastructure. We found that distributed tests had a lower flakiness rate than non-distributed tests. Interestingly, the load on the testing infrastructure was not correlated with test flakiness. The relationship between test flakiness and required resources for test execution (i.e., memory and CPU) is inconclusive.

Conclusions: Based on our findings, we conclude that splitting long-running tests can be an important measure for practitioners to cope with test flakiness. Test splitting enables parallelization of test executions and also reduces the cost of re-executions after flaky failures because the scope of the re-executed tests is narrower. Thus, splitting long-running tests into smaller tests with a narrower scope can effectively decrease the negative effects of test flakiness in complex testing environments. However, when splitting long-running tests, practitioners need to consider the potential test setup overhead of test splits.