ICST 2025
Mon 31 March - Fri 4 April 2025 Naples, Italy
Fri 4 Apr 2025 14:00 - 14:15 at Aula Magna (AM) - Test Quality Chair(s): Carolin Brandt

Flaky tests pose a significant issue for software testing. A test with a non-deterministic outcome may undermine the reliability of the testing process, making tests untrustworthy. Previous research has identified test order dependency as one of the most prevalent causes of flakiness, particularly in Java and Python. However, little is known about test order dependency in JavaScript tests. This paper aims to investigate test order dependency in JavaScript projects that use Jest, a widely used JavaScript testing framework. We implemented a systematic approach to randomise tests, test suites and describe blocks and produced 10 unique test reorders for each level. We reran each order 10 times (100 reruns for each test suite/project) and recorded any changes in test outcomes. We then manually analysed each case that showed flaky outcomes to determine the cause of flakiness. We examined our detection approach on a dataset of 81 projects obtained from GitHub. Our results revealed 55 order-dependent tests across 10 projects. Most order-dependent tests (52) occurred between tests, while the remaining three occurred between describe blocks. Those order-dependent tests are caused by either shared files (13) or shared mocking state (42) between tests. While sharing files is a known cause of order-dependent tests in other languages, our results underline a new cause (shared mocking state) that was not reported previously.

Fri 4 Apr

Displayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change

14:00 - 15:30
Test QualityResearch Papers / Short Papers, Vision and Emerging Results / Industry at Aula Magna (AM)
Chair(s): Carolin Brandt Delft University of Technology
14:00
15m
Talk
Detecting and evaluating order-dependent flaky tests in JavaScript
Research Papers
Negar Hashemi Massey University, Amjed Tahir Massey University, Shawn Rasheed Universal College of Learning, August Shi The University of Texas at Austin, Rachel Blagojevic Massey University
Pre-print
14:15
15m
Talk
An empirical study of web flaky tests: Understanding and unveiling DOM event interaction challenges
Research Papers
yu pei , Jeongju Sohn Kyungpook National University, Mike Papadakis University of Luxembourg
14:30
15m
Talk
On The Energy Consumption of Test Generation
Research Papers
Fitsum Kifetew Fondazione Bruno Kessler, Davide Prandi Fondazione Bruno Kessler, Angelo Susi Fondazione Bruno Kessler
Pre-print
14:45
7m
Talk
Improving the Comprehensibility of Generated Test Suites Using Test Case Clustering
Short Papers, Vision and Emerging Results
Mitchell Olsthoorn Delft University of Technology
14:52
7m
Talk
Harnessing Test Call Structures for Improved Fault Localization Effectiveness
Short Papers, Vision and Emerging Results
14:59
7m
Talk
Batch Execution of Microbenchmarks for Efficient Performance Testing
Short Papers, Vision and Emerging Results
Mostafa Jangali Concordia University, Kundi Yao University of Waterloo, Yiming Tang Rochester Institute of Technology, Diego Elias Costa Concordia University, Canada, Weiyi Shang University of Waterloo
15:06
15m
Talk
A Tale from the Trenches: Applying Metamorphic and Differential Testing to Bioinformatics Software
Industry
Alexis L. Marsh Iowa State University, Myra Cohen Iowa State University, Robert Cottingham Oak Ridge National Laboratory
:
:
:
: