Can We Trust the Actionable Guidance from Explainable AI Techniques in Defect Prediction?
Despite advances in high-performance Software Defect Prediction (SDP) models, practitioners remain hesitant to adopt them due to opaque decision-making and a lack of actionable insights. Recent research has applied various explainable AI (XAI) techniques to provide explainable and actionable guidance for SDP results to address these limitations, but the trustworthiness of such guidance for practitioners has not been sufficiently investigated. Practitioners may question the feasibility of implementing the proposed changes, and if these changes fail to resolve predicted defects or prove inaccurate, their trust in the guidance may diminish. In this study, we empirically evaluate the effectiveness of current XAI approaches for SDP across 32 releases of 9 large-scale projects, focusing on whether the guidance meets practitioners’ expectations. Our findings reveal that their actionable guidance (i) does not guarantee that predicted defects are resolved; (ii) fails to pinpoint modifications required to resolve predicted defects; and (iii) deviates from the typical code changes practitioners make in their projects. These limitations indicate that the guidance is not yet reliable enough for developers to justify investing their limited debugging resources. We suggest that future XAI research for SDP incorporate feedback loops that offer clear rewards for practitioners’ efforts, and propose a potential alternative approach utilizing counterfactual explanations.
Thu 6 MarDisplayed time zone: Eastern Time (US & Canada) change
14:00 - 15:30 | Defect Prediction & AnalysisResearch Papers / Industrial Track / Journal First Track at L-1710 Chair(s): Rrezarta Krasniqi University of North Carolina at Charlotte | ||
14:00 15mTalk | An ensemble learning method based on neighborhood granularity discrimination index and its application in software defect prediction Research Papers Yuqi Sha College of Information Science and Technology,Qingdao University of Science and Technology, Feng Jiang College of Information Science and Technology,Qingdao University of Science and Technology, Qiang Hu College of Information Science and Technology, Qingdao University of Science and technology, Yifan He Institute of Cosmetic Regulatory Science,Beijing Technology and Business University | ||
14:15 15mTalk | ALOGO: A Novel and Effective Framework for Online Cross-project Defect Prediction Research Papers Rongrong Shi Beijing Jiaotong University, Yuxin He Beijing Jiaotong University, Ying Liu Beijing Jiaotong University, Zonghao Li Beijing Jiaotong University, Jingxin Su Beijing Jiaotong University, Haonan Tong Beijing Jiaotong University | ||
14:30 15mTalk | Cross-System Software Log-based Anomaly Detection Using Meta-Learning Research Papers Yuqing Wang University of Helsinki, Finland, Mika Mäntylä University of Helsinki and University of Oulu, Jesse Nyyssölä University of Helsinki, Ke Ping University of Helsinki, Liqiang Wang University of Wyoming Pre-print | ||
14:45 15mTalk | RADICE: Causal Graph Based Root Cause Analysis for System Performance Diagnostic Industrial Track Andrea Tonon Huawei Ireland Research Center, Meng Zhang Shandong University, Bora Caglayan Huawei Ireland Research Center, Fei Shen Huawei Nanjing Research Center, Tong Gui , Mingxue Wang Huawei Ireland Research Center, Rong Zhou | ||
15:00 15mTalk | Can We Trust the Actionable Guidance from Explainable AI Techniques in Defect Prediction? Research Papers | ||
15:15 15mTalk | Making existing software quantum safe: A case study on IBM Db2 Journal First Track Lei Zhang University of Maryland Baltimore County, Andriy Miranskyy Toronto Metropolitan University (formerly Ryerson University), Walid Rjaibi IBM Canada Lab, Greg Stager IBM Canada Lab, Michael Gray IBM, John Peck IBM |