Predicting the Root Cause of Flaky Tests Based on Test Smells
Flaky tests refer to test cases that exhibit inconsistent behaviors across multiple executions, potentially passing or failing unpredictably. They are frequently associated with suboptimal design practices that testers may utilize when crafting test cases, which undermine the quality of software testing. Identifying the root causes of flaky tests is crucial for fixing them. Currently, inspired by the success of the Large Language Models (LLMs), researchers leverage the pre-trained language model to embed flaky test code as vectors and predict its root cause category based on vector similarity measures. However, such code embeddings generated by LLM mainly focus on capturing general semantic features but lack sufficient comprehension of the behavioral patterns involved in test scenarios, leading to the ineffectiveness of root cause identification. Test smells, which reflect poor coding practices or habits when writing test cases, provide complementary information in root cause identification of test flakiness. Therefore, this paper proposes a flaky test root cause identification method based on test smells, which leverages test smells to abstract and express behavioral patterns of test codes and integrates general semantic features extracted via vector embeddings to enhance the feature representation of flaky tests. Furthermore, to capture the complex nonlinear relationships between test smell features and code embeddings, a Feedforward Neural Network is constructed to categorize the root causes of test flakiness. To validate the effectiveness of our method, we performed evaluations on a dataset consisting of 451 Java flaky test cases. The experimental results indicate that our method achieves an F1-score of 80%, which is 7% higher than that of the baseline model that does not incorporate test smells.
Predicting the Root Cause of Flaky Tests Based on Test Smells (Predicting_the_Root_Cause_of_Flaky_Tests_Based_on_Test_Smells.pdf) | 955KiB |
Sun 27 AprDisplayed time zone: Eastern Time (US & Canada) change
16:00 - 17:30 | |||
16:00 30mPaper | DockInsight: A Knowledge-Augmented Dependency Extraction Approach for Dockerfile ICSR Zhiling Zhu Zhejiang University of Technology, Tieming Chen Zhejiang University of Technology, Yunjin Zhong Zhejiang University of Technology, Qijie Song Zhejiang University of Technology | ||
16:30 15mPaper | Porting an LLM based Application from ChatGPT to an On-Premise Environment ICSR Teemu Paloniemi University of Jyväskylä, Manu Setälä Solita Oy, Tommi Mikkonen University of Jyvaskyla Pre-print | ||
16:45 30mPaper | Predicting the Root Cause of Flaky Tests Based on Test Smells ICSR Jing Wang College of Information Science and Technology, Beijing University of Chemical Technology, Weixi Zhang College of Information Engineering, Beijing Institute of Petrochemical Technology, Beijing, China, Weixi Zhang College of Information Engineering, Beijing Institute of Petrochemical Technology, Beijing, China, Ruilian Zhao Beijing University of Chemical Technology, Ying Shang Beijing University of Chemical Technology File Attached | ||
17:15 15mPaper | Towards Patterns for a Reference Assurance Case for Autonomous Inspection Robots ICSR Dhaminda B. Abeywickrama Department of Computer Science, The University of Manchester, UK, Michael Fisher University of Manchester, UK, Frederic Wheeler Regulatory Support Directorate, Amentum, Louise Dennis The University of Manchester |