ICSE 2024
Fri 12 - Sun 21 April 2024 Lisbon, Portugal

Automated debugging is an emerging research field that aims to automatically find and repair bugs. In this field, Fault Localization (FL) and Automated Program Repair (APR) gain the most research efforts. Most recently, researchers have adopted pre-trained Large Language Models (LLMs) to facilitate FL and APR and their results are promising. However, the LLMs they used either vanished (such as Codex) or outdated (such as early versions of GPT). In this paper, we evaluate the performance of recent commercial closed-source general-purpose LLMs on FL and APR, i.e., ChatGPT 3.5, ERNIE Bot 3.5, and IFlytek Spark 2.0. We select three popular LLMs and evaluate them on 120 real-world Java bugs from the benchmark Defects4J. For FL and APR, we designed three kinds of prompts for each, considering different kinds of information. The results show that these LLMs could successfully locate 53.3% and correctly fix 12.5% of these bugs.

Sat 20 Apr

Displayed time zone: Lisbon change

14:00 - 15:30
Session 3: Keynote 2 + Position PapersLLM4Code at Luis de Freitas Branco
Chair(s): Lingming Zhang University of Illinois at Urbana-Champaign
14:00
50m
Keynote
Open development of Large Language Models for code with BigCode and StarCoder2
LLM4Code
Loubna Ben Allal Hugging Face
14:50
8m
Talk
Benchmarking the Security Aspect of Large Language Model-Based Code Generation
LLM4Code
Cheng Cheng Concordia University, Jinqiu Yang Concordia University
Pre-print
14:58
8m
Talk
Enhancing LLM-Based Coding Tools through Native Integration of IDE-Derived Static Context
LLM4Code
Yichen LI The Chinese University of Hong Kong, Yun Peng The Chinese University of Hong Kong, Yintong Huo The Chinese University of Hong Kong, Michael Lyu The Chinese University of Hong Kong
Pre-print
15:06
8m
Talk
Evaluating Fault Localization and Program Repair Capabilities of Existing Closed-Source General-Purpose LLMs
LLM4Code
Shengbei Jiang Beijing Jiaotong University, Jiabao Zhang Beijing Jiaotong University, Wei Chen Beijing Jiaotong University, Bo Wang Beijing Jiaotong University, Jianyi Zhou Huawei Cloud Computing Technologies Co., Ltd., Jie M. Zhang King's College London
Pre-print
15:14
8m
Talk
MoonBit: Explore the Design of an AI-Friendly Programming Language
LLM4Code
Haoxiang Fei International Digital Economy Academy, Yu Zhang International Digital Economy Academy, Hongbo Zhang International Digital Economy Academy, Yanlin Wang Sun Yat-sen University, Qing Liu International Digital Economy Academy
Pre-print
15:22
8m
Talk
Toward a New Era of Rapid Development: Assessing GPT-4-Vision's Capabilities in UML-Based Code Generation
LLM4Code
Gabor Antal University of Szeged, Richárd Vozár Department of Software Engineering, University of Szeged, Hungary, Rudolf Ferenc University of Szeged