ICST 2023
Sun 16 - Thu 20 April 2023 Dublin, Ireland
Wed 19 Apr 2023 15:20 - 15:40 at Pearse suite - Session 15: Flaky Tests Chair(s): John Micco

Test flakiness forms a major testing concern. Flaky tests manifest non-deterministic outcomes that cripple continuous integration and lead developers to investigate false alerts. Industrial reports indicate that on a large scale, the accrual of flaky tests breaks the trust in test suites and entails significant computational cost. To alleviate this, practitioners are constrained to identify flaky tests and investigate their impact. To shed light on such mitigation mechanisms, we interview 14 practitioners with the aim to identify (i) the sources of flakiness within the testing ecosystem, (ii) the impacts of flakiness, (iii) the measures adopted by practitioners when addressing flakiness, and (iv) the automation opportunities for these measures. Our analysis shows that, besides the tests and code, flakiness stems from interactions between the system components, the testing infrastructure, and external factors. We also highlight the impact of flakiness on testing practices and product quality and show that the adoption of guidelines together with a stable infrastructure are key measures in mitigating the problem.

Wed 19 Apr

Displayed time zone: Dublin change

14:00 - 15:40
Session 15: Flaky TestsPrevious Editions / Research Papers at Pearse suite
Chair(s): John Micco VMware
14:00
20m
Talk
Evaluating Features for Machine Learning Detection of Order- and Non-Order-Dependent Flaky Tests
Previous Editions
Owain Parry The University of Sheffield, Gregory Kapfhammer Allegheny College, Michael Hilton Carnegie Mellon University, Phil McMinn University of Sheffield
DOI
14:20
20m
Talk
An Empirical Study of Flaky Tests in Python
Previous Editions
Martin Gruber BMW Group, University of Passau, Stephan Lukasczyk University of Passau, Florian Kroiß , Gordon Fraser University of Passau
DOI
14:40
20m
Talk
A Survey on How Test Flakiness Affects Developers and What Support They Need To Address It
Previous Editions
Martin Gruber BMW Group, University of Passau, Gordon Fraser University of Passau
DOI
15:00
20m
Talk
Practical Flaky Test Prediction using Common Code Evolution and Test History Data
Research Papers
Martin Gruber BMW Group, University of Passau, Michael Heine BMW Group; Friedrich-Alexander Universität Erlangen-Nürnberg (FAU), Programming Systems Group, Norbert Oster Friedrich-Alexander Universität Erlangen-Nürnberg (FAU), Programming Systems Group, Michael Philippsen Friedrich-Alexander Universität Erlangen-Nürnberg (FAU), Programming Systems Group, Gordon Fraser University of Passau
Pre-print
15:20
20m
Talk
A Qualitative Study on the Sources, Impacts, and Mitigation Strategies of Flaky Tests
Previous Editions
Sarra Habchi Ubisoft, Guillaume Haben University of Luxembourg, Mike Papadakis University of Luxembourg, Luxembourg, Maxime Cordy University of Luxembourg, Luxembourg, Yves Le Traon University of Luxembourg, Luxembourg
DOI