LLM4SZZ: Enhancing SZZ Algorithm with Context-Enhanced Assessment on Large Language Models
The SZZ algorithm is the dominant technique for identifying bug-inducing commits and serves as a foundation for many software engineering studies, such as bug prediction and static code analysis, thereby enhancing software quality and facilitating better maintenance practices. Researchers have proposed many variants to enhance the SZZ algorithm’s performance since its introduction. The majority of them rely on static techniques or heuristic assumptions, making them easy to implement, but their performance improvements are often limited. Recently, a deep learning-based SZZ algorithm has been introduced to enhance the original SZZ algorithm performance. However, it requires complex preprocessing and is restricted to a single programming language. Additionally, while it enhances precision, it sacrifices recall. Furthermore, most of variants overlook crucial information, such as commit messages and patch context, and are limited to bug-fixing commits involving deleted lines.
The emergence of large language models (LLMs) offers an opportunity to address these drawbacks. In this study, we investigate the strengths and limitations of LLMs and propose LLM4SZZ, which employs two approaches (i.e., rank-based identification and context-enhanced identification) to handle different types of bug-fixing commits. We determine which approach to adopt based on the LLM’s ability to comprehend the bug and identify whether the bug is present in a commit. The context-enhanced identification provides the LLM with more context and requires it to find the bug-inducing commit among a set of candidate commits. In rank-based identification, we ask the LLM to select buggy statements from the bug-fixing commit and rank them based on their relevance to the root cause. Experimental results show that LLM4SZZ outperforms all baselines across three datasets, improving F1-score by 8.9% to 16.4% without significantly sacrificing recall. Additionally, LLM4SZZ can identify many bug-inducing commits that the baselines fail to detect, accounting for 7.8%, 7.4% and 2.5% of the total bug-inducing commits across three datasets, respectively.
Thu 26 JunDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
11:00 - 12:15 | Bugs and Repository MiningResearch Papers at Cosmos 3B Chair(s): Shiyi Wei University of Texas at Dallas | ||
11:00 25mTalk | LLM4SZZ: Enhancing SZZ Algorithm with Context-Enhanced Assessment on Large Language Models Research Papers Lingxiao Tang Zhejiang University, Jiakun Liu Singapore Management University, Zhongxin Liu Zhejiang University, Xiaohu Yang Zhejiang University, Lingfeng Bao Zhejiang University DOI | ||
11:25 25mTalk | Walls Have Ears: Demystifying Notification Listener Usage in Android Apps Research Papers Jiapeng Deng Huazhong University of Science and Technology, Tianming Liu Monash Univerisity, Yanjie Zhao Huazhong University of Science and Technology, Chao Wang University of Science and Technology of China, Lin Zhang The National Computer Emergency Response Team/Coordination Center of China (CNCERT/CC), Haoyu Wang Huazhong University of Science and Technology DOI | ||
11:50 25mTalk | An Investigation on Numerical Bugs in GPU Programs Towards Automated Bug Detection Research Papers Ravishka Shemal Rathnasuriya The University of Texas - Dallas, Nidhi Majoju University of Texas at Dallas, Zihe Song University of Texas at Dallas, Wei Yang UT Dallas DOI | ||
Cosmos 3B is the second room in the Cosmos 3 wing.
When facing the main Cosmos Hall, access to the Cosmos 3 wing is on the left, close to the stairs. The area is accessed through a large door with the number “3”, which will stay open during the event.