Exploring the relationship between performance metrics and cost saving potential of defect prediction models
\textit{Context:} Performance metrics are a core component of the evaluation of any machine learning model and used to compare models and estimate their usefulness. Recent work started to question the validity of many performance metrics for this purpose in the context of software defect prediction.
\textit{Objective:} Within this study, we explore the relationship between performance metrics and the cost saving potential of defect prediction models. We study whether performance metrics are suitable proxies to evaluate the cost saving capabilities and derive a theory for the relationship between performance metrics and cost saving potential.
\textit{Methods:} We measure performance metrics and cost saving potential in defect prediction experiments. We use a multinomial logit model, decision, and random forest to model the relationship between the metrics and the cost savings.
\textit{Results:} We could not find a stable relationship between cost savings and performance metrics. We attribute the lack of the relationship to the inability of performance metrics to account for the property that a small proportion of very large software artifacts are the main driver of the costs.
\textit{Conclusion:} Any defect prediction study interested in finding the best prediction model, must consider cost savings directly, because no reasonable claims regarding the economic benefits of defect prediction can be made otherwise.
Wed 17 MayDisplayed time zone: Hobart change
13:45 - 15:15 | Defect analysisJournal-First Papers / Technical Track / SEIP - Software Engineering in Practice at Meeting Room 106 Chair(s): Kla Tantithamthavorn Monash University | ||
13:45 15mTalk | RepresentThemAll: A Universal Learning Representation of Bug Reports Technical Track Sen Fang Macau University of Science and Technology, Tao Zhang Macau University of Science and Technology, Youshuai Tan Macau University of Science and Technology, He Jiang Dalian University of Technology, Xin Xia Huawei, Xiaobing Sun Yangzhou University | ||
14:00 15mTalk | Demystifying Exploitable Bugs in Smart Contracts Technical Track Zhuo Zhang Purdue University, Brian Zhang Harrison High School (Tippecanoe), Wen Xu PNM Labs, Zhiqiang Lin The Ohio State University Pre-print | ||
14:15 15mTalk | Understanding and Detecting On-the-Fly Configuration Bugs Technical Track Teng Wang National University of Defense Technology, Zhouyang Jia National University of Defense Technology, Shanshan Li National University of Defense Technology, Si Zheng National University of Defense Technology, Yue Yu College of Computer, National University of Defense Technology, Changsha 410073, China, Erci Xu National University of Defense Technology, Shaoliang Peng Hunan University, Liao Xiangke National University of Defense Technology Pre-print | ||
14:30 15mTalk | Explaining Software Bugs Leveraging Code Structures in Neural Machine Translation Technical Track Parvez Mahbub Dalhousie University, Ohiduzzaman Shuvo Dalhousie University, Masud Rahman Dalhousie University Pre-print Media Attached | ||
14:45 15mTalk | Scalable Compositional Static Taint Analysis for Sensitive Data Tracing on Industrial Micro-Services SEIP - Software Engineering in Practice Zexin Zhong Ant Group; University of Technology Sydney, Jiangchao Liu Ant Group, Diyu Wu Ant Group, Peng Di Ant Group, Yulei Sui University of New South Wales, Sydney, Alex X. Liu Ant Group, John C.S. Lui The Chinese University of Hong Kong | ||
15:00 7mTalk | Exploring the relationship between performance metrics and cost saving potential of defect prediction models Journal-First Papers | ||
15:07 7mTalk | A Machine and Deep Learning analysis among SonarQube rules, Product, and Process Metrics for Faults Prediction Journal-First Papers Francesco Lomio Constructor Institute Schaffhausen, Sergio Moreschini Tampere University, Valentina Lenarduzzi University of Oulu | ||