Pinpointing Flakiness in Web Tests via DOM Event Analysis and LLM-based Reasoning
Flaky tests in web applications frequently arise from nondeterministic interactions between asynchronous events and dynamic DOM updates. Traditional debugging techniques, including fault localization, struggle to diagnose such failures due to the complex, event-driven, and continuously evolving nature of modern web interfaces. This paper introduces DoeFL, a novel approach that combines structured analysis of DOM event sequences with large language model (LLM) reasoning to identify the root causes of web flakiness. DoeFL models temporal causal relationships between DOM events and flaky test failures using Lamport logical clocks. It then uses LLMs to capture semantic inconsistencies and intent mismatches often missed by purely structural analyses. We evaluate DoeFL on 122 real-world flaky test cases collected from 47 web projects. Using only its Lamport-based structural component, DoeFL achieves 47.5% Top-1 and 82.0% Top-5 localization accuracy. When augmented with LLM-based reasoning, performance further improves, achieving Top-1 accuracy between 65.6% and 77.9% and Top-5 accuracy between 89.3% and 95.1%,= depending on the selected LLM.
Tue 19 MayDisplayed time zone: Seoul change
14:00 - 15:30 | Test FlakinessResearch Papers / Short Papers, Vision and Emerging Results at Room 101 Chair(s): Phil McMinn University of Sheffield | ||
14:00 25mTalk | A Systematic Evaluation of Environmental Flakiness in JavaScript Tests Research Papers Negar Hashemi Massey University, Amjed Tahir Massey University, August Shi The University of Texas at Austin, Shawn Rasheed UCOL, Rachel Blagojevic Massey University | ||
14:25 25mTalk | Pinpointing Flakiness in Web Tests via DOM Event Analysis and LLM-based Reasoning Research Papers Yu Pei University of Luxembourg, Jeongju Sohn Kyungpook National University, Sarra Habchi Cohere, Canada, Mike Papadakis University of Luxembourg | ||
14:50 25mTalk | Automatically Resolving Mismatched Invocations on Test Doubles in Test Suites Research Papers | ||
15:15 15mTalk | Towards an Actionable Flakiness Score Short Papers, Vision and Emerging Results Joanna Kisaakye University of Antwerp, Mutlu Beyazıt University of Antwerp and Flanders Make vzw, Serge Demeyer University of Antwerp and Flanders Make vzw | ||