2nd International Flaky Tests Workshop 2025 (FTW 2025)FTW 2025
Tool and Dataset Demo
We are planning on running a session of tool and dataset demonstrations at the workshop. If you are planning to attend FTW and would like to present at this session, please fill in this Google form: https://docs.google.com/forms/d/e/1FAIpQLScC17IP_cCFSvwbccm1yTpZntjDTC247u4U2LxyZ9nawtwpNg/viewform?usp=sharing.
What is a flaky test?
Software developers rely on test cases to identify bugs in their code and to provide a signal as to their code’s correctness. Should such signals have a history of unreliability, they not only become less informative, but may also be considered untrustworthy. In the context of software testing, practitioners refer to these unreliable signals as flaky tests. The definition varies slightly, but a flaky test is generally defined as a test case that can pass and fail without changes to the test case code or the code under test.
Why are they such a big deal?
Concurrency and randomness are well-established causes among many others, though flakiness has far-reaching negative consequences regardless of origin. These consequences are felt by developers from small open-source projects to the likes of Google, Microsoft, and Meta. Flaky tests challenge the assumption that a test failure implies a bug, constituting a leading cause of “false alarm” test failures, and potentially more seriously, having the potential to mask the presence of a genuine bug. Flaky tests may lead to time wasted debugging spurious failures, leading developers to ignore future test failures. This is detrimental to software stability, because while a flaky test may be unreliable, it could still indicate a genuine bug in some instances. This is further exacerbated when flaky tests accumulate, as developers may lose trust in the entire test suite.
What are we going to do about it?
Flaky tests as a research topic has grown in interest significantly within the software engineering community in recent years. This has produced a wide array of empirical studies on the causes of flaky tests and experimental tools for their detection and repair. We are delighted to announce the return of the International Flaky Test Workshop (FTW) for a second year at ICSE 2025. The workshop welcomes submissions on topics relating to flaky tests and will provide an opportunity for academic researchers and industrial practitioners to exchange ideas about test flakiness. Please see the Call for Papers for more information.
This program is tentative and subject to change.
Sun 27 AprDisplayed time zone: Eastern Time (US & Canada) change
07:00 - 19:00 | Ready Room SundaySocial, Networking and Special Rooms at 209 The Ready Room will be available throughout the week. There will be some tables with computers where people can edit presentations (bring on a USB stick) and upload presentations to the presentation rooms through the Contact 1 website. There will also be AV technicians to help if needed. You do not need to use the Ready Room: You have several choices: You can upload your presentation from your own computer in advance of your session (days in advance even) at the Contact 1 website (you will be sent a link). Or you can plug your computer in using an HDMI cable when you are starting your presentation. This last option is available but not recommended, since it increases the chance of delays. There will be some tables and couches in the Ready Room where you can get work done, or have small get-togethers with people. This room will not be ‘quiet’. If you want a quiet place to work or chill out (library quiet, no talking) then Room 209 will be available much of the time. The Ready Room will also have some poster boards. | ||
09:00 - 12:30 | Child Care Sunday AMSocial, Networking and Special Rooms at 102 Child Care Child Care at ICSE is free, but you must have registered for child care when you registered for the conference. If you need to add child care to your registration, please contact the registration desk. | ||
09:00 - 10:30 | |||
09:00 90mKeynote | Keynote FTW Michael Hilton Carnegie Mellon University | ||
10:30 - 11:00 | |||
10:30 30mCoffee break | Sunday Morning Break Catering | ||
11:00 - 12:30 | |||
11:00 22mPaper | A Preliminary Study of Fixed Flaky Tests in Rust Projects on GitHub FTW Tom Schroeder University of Illinois Urbana-Champaign, Minh Phan University of Illinois Urbana-Champaign, Yang Chen University of Illinois at Urbana-Champaign | ||
11:22 22mTalk | Beyond Test Flakiness: A Manifesto for a Holistic Approach to Test Suite Health FTW Phil McMinn University of Sheffield, Muhammad Firhard Roslan University of Sheffield, Gregory Kapfhammer Allegheny College | ||
11:45 22mPaper | Empirically Evaluating Flaky Tests for Autonomous Driving Systems in Simulated Environments FTW Olek Osikowicz University of Sheffield, UK, Phil McMinn University of Sheffield, Donghwan Shin University of Sheffield | ||
12:07 22mPanel | Mini Panel 1 FTW | ||
12:30 - 14:00 | |||
12:30 90mLunch | Sunday Lunch Catering | ||
14:00 - 15:30 | |||
14:00 22mPaper | Evaluating NonDex for Modern Java Ecosystem FTW Kaiyao Ke University of Illinois at Urbana-Champaign, Darko Marinov University of Illinois at Urbana-Champaign | ||
14:22 22mTalk | On the Perception and Adoption of Flakiness Scores FTW Joanna Kisaakye , Mutlu Beyazıt University of Antwerp and Flanders Make vzw, Serge Demeyer University of Antwerp and Flanders Make vzw | ||
14:45 22mPaper | Reduction of Test Re-runs by Prioritizing Potential Order Dependent Flaky Tests FTW Hasnain Iqbal University of Dhaka, Zerina Begum University of Dhaka, Kazi Sakib Institute of Information Technology, University of Dhaka | ||
15:07 22mPanel | Mini Panel 2 FTW | ||
15:30 - 16:00 | |||
15:30 30mBreak | Sunday Afternoon Break Catering | ||
16:00 - 17:30 | |||
16:00 90mDemonstration | Dataset/Tool Demonstrations FTW | ||
Accepted Papers
Call for Papers
FTW welcomes submissions on topics relating to flaky tests and non-determinism in testing generally. The workshop will provide an opportunity for academic researchers and industrial practitioners to exchange ideas about test flakiness and to find out about current research directions and industrial challenges. A major goal of FTW is to foster collaboration and exchange between academia and industry. The workshop is inclusive of quantitative, qualitative, and mixed-methods research. Topics of interest include (but are not limited to):
- Causes of flaky tests.
- Costs and consequences of flaky tests.
- Debugging of flaky tests.
- Detection of flaky tests.
- Mitigation of flaky tests.
- Non-determinism in testing generally.
- Repair of flaky tests.
We expect a significant portion of the day to be spent on presentations and discussions of extended abstracts, but there will also be more formal short paper presentations. Please note that due to ICSE restrictions, submissions cannot exceed 8 pages. Submissions can take one of two formats:
- Extended abstract (max. 2 pages including references): New ideas, problems and challenges, view points, work in progress.
- Short paper (max. 8 pages including references): Technical research, experience reports, empirical studies.
Submission
All submissions must be made via the following link: https://easychair.org/my/conference?conf=ftw2025.
Each submission will be reviewed by the program committee with respect to suitability for the workshop, following a double-blind process for short papers and a single-blind process for extended abstracts. This means that the identity of short paper authors must not be revealed in their submissions. Submissions must conform to the IEEE conference proceedings template, specified in the IEEE Conference Proceedings Formatting Guidelines (title in 24pt font and full text in 10pt type, LaTeX users must use \documentclass[10pt,conference]{IEEEtran} without including the compsoc or compsocconf options). Note that IEEE format is being used this year, whereas last year it was ACM format, hence the appearance will differ from year to year.
Important Dates
- Paper submission: November 11th 2024 AoE.
- Acceptance notification: December 1st 2024 AoE.
- Camera ready: February 5th 2025 AoE.
| Wed 5 Feb 2025 Camera ready |
| Sun 1 Dec 2024 Acceptance notification |
| Mon 11 Nov 2024 Paper submission |
Sebastian Baltes
University of Bayreuth
Germany
Filomena Ferrucci
University of Salerno
Italy
Guillaume Haben
University of Luxembourg
Hadi Hemmati
York University
Canada
Michael Hilton
Carnegie Mellon University
United States
Gregory Kapfhammer
Allegheny College
United States
Darko Marinov
University of Illinois at Urbana-Champaign
United States
Valeria Pontillo
Vrije Universiteit Brussel
Belgium
Shanto Rahman
The University of Texas at Austin
United States
Marcelo d'Amorim
North Carolina State University
United States