Evaluating Fault Localization and Program Repair Capabilities of Existing Closed-Source General-Purpose LLMs
Automated debugging is an emerging research field that aims to automatically find and repair bugs. In this field, Fault Localization (FL) and Automated Program Repair (APR) gain the most research efforts. Most recently, researchers have adopted pre-trained Large Language Models (LLMs) to facilitate FL and APR and their results are promising. However, the LLMs they used either vanished (such as Codex) or outdated (such as early versions of GPT). In this paper, we evaluate the performance of recent commercial closed-source general-purpose LLMs on FL and APR, i.e., ChatGPT 3.5, ERNIE Bot 3.5, and IFlytek Spark 2.0. We select three popular LLMs and evaluate them on 120 real-world Java bugs from the benchmark Defects4J. For FL and APR, we designed three kinds of prompts for each, considering different kinds of information. The results show that these LLMs could successfully locate 53.3% and correctly fix 12.5% of these bugs.
Sat 20 AprDisplayed time zone: Lisbon change
14:00 - 15:30 | Session 3: Keynote 2 + Position PapersLLM4Code at Luis de Freitas Branco Chair(s): Lingming Zhang University of Illinois at Urbana-Champaign | ||
14:00 50mKeynote | Open development of Large Language Models for code with BigCode and StarCoder2 LLM4Code Loubna Ben Allal Hugging Face | ||
14:50 8mTalk | Benchmarking the Security Aspect of Large Language Model-Based Code Generation LLM4Code Pre-print | ||
14:58 8mTalk | Enhancing LLM-Based Coding Tools through Native Integration of IDE-Derived Static Context LLM4Code Yichen LI The Chinese University of Hong Kong, Yun Peng The Chinese University of Hong Kong, Yintong Huo The Chinese University of Hong Kong, Michael Lyu The Chinese University of Hong Kong Pre-print | ||
15:06 8mTalk | Evaluating Fault Localization and Program Repair Capabilities of Existing Closed-Source General-Purpose LLMs LLM4Code Shengbei Jiang Beijing Jiaotong University, Jiabao Zhang Beijing Jiaotong University, Wei Chen Beijing Jiaotong University, Bo Wang Beijing Jiaotong University, Jianyi Zhou Huawei Cloud Computing Technologies Co., Ltd., Jie M. Zhang King's College London Pre-print | ||
15:14 8mTalk | MoonBit: Explore the Design of an AI-Friendly Programming Language LLM4Code Haoxiang Fei International Digital Economy Academy, Yu Zhang International Digital Economy Academy, Hongbo Zhang International Digital Economy Academy, Yanlin Wang Sun Yat-sen University, Qing Liu International Digital Economy Academy Pre-print | ||
15:22 8mTalk | Toward a New Era of Rapid Development: Assessing GPT-4-Vision's Capabilities in UML-Based Code Generation LLM4Code Gabor Antal University of Szeged, Richárd Vozár Department of Software Engineering, University of Szeged, Hungary, Rudolf Ferenc University of Szeged |