Evaluating SZZ Implementations: An Empirical Study on the Linux Kernel
The SZZ algorithm is used to connect bug-fixing commits to the earlier commits that introduced bugs. This algorithm has many applications and many variants have been devised. However, there are some types of commits that cannot be traced by the SZZ algorithm, referred to as “ghost commits.” The evaluation of how these ghost commits impact the SZZ implementations remains limited. Moreover, these implementations have been evaluated on datasets created by software engineering researchers from information in bug trackers and version controlled histories.
Since Oct 2013, the Linux kernel developers have started labelling bug-fixing patches with the commit identifiers of the corresponding bug-inducing commit(s) as a standard practice. As of v6.1-rc5, 76,046 pairs of bug-fixing patches and bug-inducing commits are available. This provides a unique opportunity to evaluate the SZZ algorithm on a large dataset that has been created and reviewed by project developers, entirely independently of the biases of software engineering researchers.
In this paper, we apply six SZZ implementations to 76,046 pairs of bug-fixing patches and bug-introducing commits from the Linux kernel. Our findings reveal that SZZ algorithms experience a more significant decline in recall on our dataset (↓ 13.8%) as compared to prior findings reported by Rosa et al., and the disparities between the individual SZZ algorithms diminish. Moreover, we find that 17.47% of bug-fixing commits are ghost commits. Finally, we propose Tracing-Commit SZZ (TC-SZZ), that traces all commits in the change history of lines modified or deleted in bug-fixing commits. Applying TC-SZZ to all failure cases, excluding ghost commits, we found that TC-SZZ could identify 17.7% of them. Our further analysis based on git log found that 34.6% of bug-inducing commits were in the function history, 27.5% in the file history (but not in the function history), and 37.9% not in the file history. We further evaluated the effectiveness of ChatGPT in boosting the SZZ algorithm’s ability to identify bug-inducing commits in the function history, in the file history and not in the file history.
Mon 23 JunDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
16:00 - 18:00 | MSR 1Industry Papers / Ideas, Visions and Reflections / Research Papers / Journal First at Aurora B Chair(s): Andrew Begel Carnegie Mellon University | ||
16:00 20mTalk | On Refining the SZZ Algorithm with Bug Discussion Data Journal First Pooja Rani University of Zurich, Fernando Petrulio University of Zurich, Alberto Bacchelli University of Zurich | ||
16:20 20mTalk | SemBIC: Semantic-aware Identification of Bug-inducing Commits Research Papers Xiao Chen The Hong Kong University of Science and Technology, Hengcheng Zhu The Hong Kong University of Science and Technology, Jialun Cao Hong Kong University of Science and Technology, Ming Wen Huazhong University of Science and Technology, Shing-Chi Cheung Hong Kong University of Science and Technology DOI | ||
16:40 20mTalk | Evaluating SZZ Implementations: An Empirical Study on the Linux Kernel Journal First Yunbo Lyu Singapore Management University, Hong Jin Kang University of Sydney, Ratnadira Widyasari Singapore Management University, Singapore, Julia Lawall Inria, David Lo Singapore Management University | ||
17:00 10mTalk | HyperSeq: A Hyper-Adaptive Representation for Predictive Sequencing of States Ideas, Visions and Reflections | ||
17:10 10mTalk | LLMs for Defect Prediction in Evolving Datasets: Emerging Results and Future Directions Ideas, Visions and Reflections Umamaheswara Sharma B National Institute of Technology, Calicut, Farhan Chonari National Institute of Technology Calicut, Gokul K Anilkumar National Institute of Technology Calicut, Saikiran Konchada National Institute of Technology Calicut | ||
17:20 20mTalk | ROSE LCOM Tools Industry Papers Kenneth Lamar University of Central Florida, Peter Pirkelbauer Lawrence Livermore National Laboratory, Zachary Painter University of Central Florida, Damian Dechev University of Central Florida | ||
Aurora B is the second room in the Aurora wing.
When facing the main Cosmos Hall, access to the Aurora wing is on the right, close to the side entrance of the hotel.