Problems with with SZZ and Features: An empirical assessment of the state of practice of defect prediction data collection
\textit{Context:} The SZZ algorithm is the de facto standard for labeling bug fixing commits and finding inducing changes for defect prediction data. Recent research uncovered potential problems in different parts of the SZZ algorithm. Most defect prediction data sets provide only static code metrics as features, while research indicates that other features are also important.
\textit{Objective:} We provide an empirical analysis of the defect labels created with the SZZ algorithm and the impact of commonly used features on results.
\textit{Method:} We used a combination of manual validation and adopted or improved heuristics for the collection of defect data. We conducted an empirical study on 398 releases of 38 Apache projects.
\textit{Results:} We found that only half of the bug fixing commits determined by SZZ are actually bug fixing. If a six-month time frame is used in combination with SZZ to determine which bugs affect a release, one file is incorrectly labeled as defective for every file that is correctly labeled as defective. In addition, two defective files are missed. We also explored the impact of the relatively small set of features that are available in most defect prediction data sets, as there are multiple publications that indicate that, e.g., churn related features are important for defect prediction. We found that the difference of using more features is not significant.
\textit{Conclusion:} Problems with inaccurate defect labels are a severe threat to the validity of the state of the art of defect prediction. Small feature sets seem to be a less severe threat.
Wed 17 MayDisplayed time zone: Hobart change
11:00 - 12:30 | Mining software repositoriesTechnical Track / Journal-First Papers / DEMO - Demonstrations at Meeting Room 102 Chair(s): Brittany Johnson George Mason University | ||
11:00 15mTalk | The untold story of code refactoring customizations in practice Technical Track Daniel Oliveira PUC-Rio, Wesley Assunção Johannes Kepler University Linz, Austria & Pontifical Catholic University of Rio de Janeiro, Brazil, Alessandro Garcia PUC-Rio, Ana Carla Bibiano PUC-Rio, Márcio Ribeiro Federal University of Alagoas, Brazil, Rohit Gheyi Federal University of Campina Grande, Baldoino Fonseca Federal University of Alagoas (UFAL) Pre-print | ||
11:15 15mTalk | Data Quality for Software Vulnerability Datasets Technical Track Roland Croft The University of Adelaide, Muhammad Ali Babar University of Adelaide, M. Mehdi Kholoosi University of Adelaide Pre-print | ||
11:30 15mTalk | Do code refactorings influence the merge effort? Technical Track André Oliveira Federal Fluminense University, Vania Neves Universidade Federal Fluminense (UFF), Alexandre Plastino Federal Fluminense University, Ana Carla Bibiano PUC-Rio, Alessandro Garcia PUC-Rio, Leonardo Murta Universidade Federal Fluminense (UFF) | ||
11:45 7mTalk | ActionsRemaker: Reproducing GitHub Actions DEMO - Demonstrations Hao-Nan Zhu University of California, Davis, Kevin Guan University of California, Davis, Robert M. Furth University of California, Davis, Cindy Rubio-González University of California at Davis | ||
11:52 7mTalk | Problems with with SZZ and Features: An empirical assessment of the state of practice of defect prediction data collection Journal-First Papers Steffen Herbold University of Passau, Alexander Trautsch University of Passau, Alexander Trautsch Germany, Benjamin Ledel None | ||
12:00 7mTalk | An empirical study of issue-link algorithms: which issue-link algorithms should we use? Journal-First Papers Masanari Kondo Kyushu University, Yutaro Kashiwa Nara Institute of Science and Technology, Yasutaka Kamei Kyushu University, Osamu Mizuno Kyoto Institute of Technology | ||
12:07 7mTalk | SCS-Gan: Learning Functionality-Agnostic Stylometric Representations for Source Code Authorship Verification Journal-First Papers Weihan Ou Queen's University at Kingston, Ding Steven, H., H. Queen’s University at Kingston, Yuan Tian Queens University, Kingston, Canada, Leo Song Queen’s University at Kingston | ||
12:15 15mTalk | A Comprehensive Study of Real-World Bugs in Machine Learning Model Optimization Technical Track Hao Guan The University of Queensland, Ying Xiao Southern University of Science and Technology, Jiaying LI Microsoft, Yepang Liu Southern University of Science and Technology, Guangdong Bai University of Queensland | ||