Flaky tests that non-deterministically pass or fail are a major impediment to regression testing. Existing work on identifying flaky tests is mostly based on test re-executions, which may be a feasible solution for short-running unit tests, but can become prohibitively expensive for long-running integration and system tests. To alleviate the problem, we propose a novel heuristic for identifying flaky test results by analyzing run-length encodings of regression test histories. For flagging tests as flaky, the proposed heuristic solely requires knowledge about previous test results and neither requires test re-executions nor expensive model fitting. We apply this heuristic in the context of the CPAchecker project, which heavily relies on integration tests for regression testing.
Mon 11 SepDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
15:30 - 17:00 | |||
15:30 20mTalk | Timeout Prediction for Software Analyses [Workshop] CPAchecker Nicola Thoben University of Oldenburg | ||
15:50 20mTalk | Real-World Software Verification with CPAchecker [Workshop] CPAchecker Thomas Lemberger LMU Munich | ||
16:10 20mTalk | Using CPAchecker in Teaching [Workshop] CPAchecker Jan Haltermann University of Oldenburg File Attached | ||
16:30 20mTalk | Handling Flaky Regression Tests in CPAchecker [Workshop] CPAchecker Philipp Wendler LMU Munich |