2nd International Flaky Tests Workshop 2024 (FTW 2025)FTW 2025

What is a flaky test?

Software developers rely on test cases to identify bugs in their code and to provide a signal as to their code’s correctness. Should such signals have a history of unreliability, they not only become less informative, but may also be considered untrustworthy. In the context of software testing, practitioners refer to these unreliable signals as flaky tests. The definition varies slightly, but a flaky test is generally defined as a test case that can pass and fail without changes to the test case code or the code under test.

Why are they such a big deal?

Concurrency and randomness are well-established causes among many others, though flakiness has far-reaching negative consequences regardless of origin. These consequences are felt by developers from small open-source projects to the likes of Google, Microsoft, and Meta. Flaky tests challenge the assumption that a test failure implies a bug, constituting a leading cause of “false alarm” test failures, and potentially more seriously, having the potential to mask the presence of a genuine bug. Flaky tests may lead to time wasted debugging spurious failures, leading developers to ignore future test failures. This is detrimental to software stability, because while a flaky test may be unreliable, it could still indicate a genuine bug in some instances. This is further exacerbated when flaky tests accumulate, as developers may lose trust in the entire test suite.

What are we going to do about it?

Flaky tests as a research topic has grown in interest significantly within the software engineering community in recent years. This has produced a wide array of empirical studies on the causes of flaky tests and experimental tools for their detection and repair. We are delighted to announce the return of the International Flaky Test Workshop (FTW) for a second year at ICSE 2025. The workshop welcomes submissions on topics relating to flaky tests and will provide an opportunity for academic researchers and industrial practitioners to exchange ideas about test flakiness. Please see the Call for Papers for more information.

Call for Papers

FTW welcomes submissions on topics relating to flaky tests and non-determinism in testing generally. The workshop will provide an opportunity for academic researchers and industrial practitioners to exchange ideas about test flakiness and to find out about current research directions and industrial challenges. A major goal of FTW is to foster collaboration and exchange between academia and industry. The workshop is inclusive of quantitative, qualitative, and mixed-methods research. Topics of interest include (but are not limited to):

• Causes of flaky tests.
• Costs and consequences of flaky tests.
• Debugging of flaky tests.
• Detection of flaky tests.
• Mitigation of flaky tests.
• Non-determinism in testing generally.
• Repair of flaky tests.

We expect a significant portion of the day to be spent on presentations and discussions of extended abstracts, but there will also be more formal short paper presentations. Please note that due to ICSE restrictions, submissions cannot exceed 8 pages. Submissions can take one of two formats:

• Extended abstract (max. 2 pages including references): New ideas, problems and challenges, view points, work in progress.
• Short paper (max. 8 pages including references): Technical research, experience reports, empirical studies.

Submission

All submissions must be made via the following link: https://easychair.org/my/conference?conf=ftw2025.

Each submission will be reviewed by the program committee with respect to suitability for the workshop, following a double-blind process for short papers and a single-blind process for extended abstracts. This means that the identity of short paper authors must not be revealed in their submissions. Submissions must conform to the IEEE conference proceedings template, specified in the IEEE Conference Proceedings Formatting Guidelines (title in 24pt font and full text in 10pt type, LaTeX users must use \documentclass[10pt,conference]{IEEEtran} without including the compsoc or compsocconf options). Note that IEEE format is being used this year, whereas last year it was ACM format, hence the appearance will differ from year to year.

Important Dates

• Paper submission: November 11th 2024 AoE.
• Acceptance notification: December 1st 2024 AoE.
• Camera ready: February 5th 2025 AoE.