XAI4FL: Enhancing Spectrum-Based Fault Localization with Explainable Artificial Intelligence
Manually finding the program unit (e.g., class, method, or statement) responsible for a fault is tedious and time-consuming. To mitigate this problem, many fault localization techniques have been proposed. A popular family of such techniques is spectrum-based fault localization (SBFL), which takes program execution traces (spectra) of failed and passed test cases as input and applies a ranking formula to compute a suspiciousness score for each program unit. However, most existing SBFL techniques fail to consider two facts: 1) not all failed test cases contribute equally to a considered fault(s), and 2) program units collaboratively contribute to the failure/pass of each test case in different ways.
In this study, we propose a novel idea that first models the SBFL task as a classification problem of predicting whether a test case will fail or pass based on spectra information on program units. We subsequently apply eXplainable Artificial Intelligence (XAI) techniques to infer the local importance of each program unit to the prediction of each executed test case. Applying XAI to the failed test case, we retrieve information about which program statements within the test case that are considered the most important (i.e., have the biggest effect in making the test case failed). Such a design can automatically learn the unique contributions of failed test cases to the suspiciousness of a program unit by learning the different and collaborative contributions of program units to each test case’s executed result. As far as we know, this is the first XAI-supported SBFL approach. We evaluate the new approach on the Defects4J benchmark dataset.
We compare the performance of our approach against five popular SBFL techniques: DStar, Tarantula, Barinel, Ochiai, and OP. We measure their performance using the Top-K and EXAM scores. In particular, we focus on the result of the Top-1, which importance has been highlighted in the automated program repair domain, where the proposed methods often assume perfect fault localization (i.e., the fault must be found at the first rank of the suspiciousness list). Our results show that our approach, named XAI4FL, has a statistically significant and substantially better performance in terms of Top-1 than the SBFL approaches. We also compare our approach with a simpler approach to get feature importance in a tree-based model (i.e., using the Mean Decrease in Impurity method). Our results show that XAI4FL statistically significantly outperforms the MDI method in Top-K and EXAM score. Our results and findings highlight that the utilization of XAI for fault localization can improve the overall results of fault localization techniques.
Tue 17 MayDisplayed time zone: Eastern Time (US & Canada) change
02:00 - 02:50
|Context-based Cluster Fault Localization
Junji Yu Chongqing University, Yan Lei School of Big Data & Software Engineering, Chongqing University, Huan Xie Chongqing University, Lingfeng Fu Chongqing University, Chunyan Liu Chongqing UniversityPre-print Media Attached
|A Study of Single Statement Bugs Involving Dynamic Language Features
Early Research Achievements (ERA)
Li Sui Massey University, New Zealand, Shawn Rasheed Massey University, Amjed Tahir Massey University, Jens Dietrich Victoria University of WellingtonPre-print Media Attached
|XAI4FL: Enhancing Spectrum-Based Fault Localization with Explainable Artificial Intelligence
Ratnadira Widyasari Singapore Management University, Singapore, Gede Artha Azriadi Prana Singapore Management University, Stefanus Agus Haryono Singapore Management University, Yuan Tian Queens University, Kingston, Canada, Hafil Noer Zachiary Singapore Management University, David Lo Singapore Management UniversityPre-print
|Do Visual Issue Reports Help Developers Fix Bugs? – A Preliminary Study of Using Videos and Images to Report Issues on GitHub –
Early Research Achievements (ERA)
Hiroki Kuramoto Kyushu University, Masanari Kondo Kyushu University, Yutaro Kashiwa Kyushu University, Yuta Ishimoto Kyushu University, Kaze Shindo Kyushu University, Yasutaka Kamei Kyushu University, Naoyasu Ubayashi Kyushu UniversityMedia Attached
|Find Bugs in Static Bug Finders
Junjie Wang Institute of Software at Chinese Academy of Sciences, Yuchao Huang Institute of Software Chinese Academy of Sciences, Song Wang York University, Qing Wang Institute of Software at Chinese Academy of SciencesPre-print Media Attached
|didiffff: A Viewer for Comparing Changes in both Code and Execution Trace
Tetsuya Kanda Osaka University, Kazumasa Shimari Nara Institute of Science and Technology, Katsuro Inoue Osaka UniversityPre-print Media Attached
|Q&A-Paper Session 11