Exploring the Potential of ChatGPT in Automated Code Refinement: An Empirical Study
Code review is an essential activity for ensuring the quality and maintainability of software projects. However, it is a time-consuming and often error-prone task that can significantly impact the development process. Recently, ChatGPT, a cutting-edge language model, has demonstrated impressive performance in various natural language processing tasks, suggesting its potential to automate code review processes. However, it is still unclear how well ChatGPT performs in code review tasks. To fill this gap, in this paper, we conduct the first empirical study to understand the capabilities of ChatGPT in code review tasks, specifically focusing on automated code refinement based on given code reviews. To conduct the study, we select the existing benchmark CodeReview and construct a new code review dataset with high quality. We use CodeReviewer, a state-of-the-art code review tool, as a baseline for comparison with ChatGPT. Our results show that ChatGPT outperforms CodeReviewer in code refinement tasks. Specifically, our results show that ChatGPT achieves higher EM and BLEU scores of 22.78 and 76.44 respectively, while the state-of-the-art method achieves only 15.50 and 62.88 on a high-quality code review dataset. We further identify the root causes for ChatGPT’s underperformance and propose several strategies to mitigate these challenges. Our study provides insights into the potential of ChatGPT in automating the code review process, and highlights the potential research directions.
Thu 18 AprDisplayed time zone: Lisbon change
11:00 - 12:30 | Language Models and Generated Code 2Demonstrations / Research Track at Maria Helena Vieira da Silva Chair(s): Reyhaneh Jabbarvand University of Illinois at Urbana-Champaign | ||
11:00 15mTalk | Exploring the Potential of ChatGPT in Automated Code Refinement: An Empirical Study Research Track Qi Guo Tianjin University, China, Junming Cao Fudan University, Xiaofei Xie Singapore Management University, Shangqing Liu Nanyang Technological University, Xiaohong Li Tianjin University, Bihuan Chen Fudan University, Xin Peng Fudan University | ||
11:15 15mTalk | Deep Learning or Classical Machine Learning? An Empirical Study on Log-Based Anomaly Detection Research Track BoXi Yu The Chinese University of Hong Kong, Shenzhen, Jiayi Yao The Chinese University of Hong Kong, Shenzhen, Qiuai Fu Huawei Cloud Computing Technologies CO., LTD., Zhiqing Zhong Chinese University of Hong Kong, Shenzhen, Haotian Xie The Chinese University of Hong Kong, Shenzhen, Yaoliang Wu Huawei Cloud Computing Technologies Co., Ltd., Yuchi Ma Huawei Cloud Computing Technologies CO., LTD., Pinjia He Chinese University of Hong Kong, Shenzhen | ||
11:30 15mTalk | TRACED: Execution-aware Pre-training for Source Code Research Track Yangruibo Ding Columbia University, Benjamin Steenhoek Iowa State University, Kexin Pei The University of Chicago, Gail Kaiser Columbia University, Wei Le Iowa State University, Baishakhi Ray AWS AI Labs | ||
11:45 15mTalk | On Extracting Specialized Code Abilities from Large Language Models: A Feasibility Study Research Track Li Zongjie Hong Kong University of Science and Technology, Chaozheng Wang The Chinese University of Hong Kong, Pingchuan Ma HKUST, Chaowei Liu National University of Singapore, Shuai Wang The Hong Kong University of Science and Technology, Daoyuan Wu Nanyang Technological University, Cuiyun Gao Harbin Institute of Technology, Yang Liu Nanyang Technological University | ||
12:00 15mTalk | When Neural Code Completion Models Size up the Situation: Attaining Cheaper and Faster Completion through Dynamic Model Inference Research Track Zhensu Sun Singapore Management University, Xiaoning Du Monash University, Australia, Fu Song State Key Laboratory of Computer Science and Institute of Software, Chinese Academy of Sciences., Shangwen Wang National University of Defense Technology, Li Li Beihang University Pre-print | ||
12:15 7mTalk | TestSpark: IntelliJ IDEA’s Ultimate Test Generation Companion Demonstrations Arkadii Sapozhnikov JetBrains Research, Mitchell Olsthoorn Delft University of Technology, Annibale Panichella Delft University of Technology, Vladimir Kovalenko JetBrains Research, Pouria Derakhshanfar JetBrains Research | ||