Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues
Since its introduction in November 2022, ChatGPT has rapidly gained popularity due to its remarkable ability in language understanding and human-like responses. ChatGPT, based on GPT-3.5 architecture, has shown great promise for revolutionizing various research fields, including code generation. However, the reliability and quality of code generated by ChatGPT remain unexplored, raising concerns about potential risks associated with the widespread use of ChatGPT-driven code generation.
In this paper, we systematically study the quality of 4,066 ChatGPT-generated code implemented in two popular programming languages, i.e., Java and Python, for 2,033 programming tasks. The goal of this work is three folds. First, we analyze the correctness of ChatGPT on code generation tasks and uncover the factors that influence its effectiveness, including task difficulty, programming language, time that tasks are introduced, and program size. Second, we identify and characterize potential issues with the quality of ChatGPT-generated code. Last, we provide insights into how these issues can be mitigated. Experiments highlight that out of 4,066 programs generated by ChatGPT, 2,756 programs are deemed correct, 1,082 programs provide wrong outputs, and 177 programs contain compilation or runtime errors. Additionally, we further analyze other characteristics of the generated code through static analysis tools, such as code style and maintainability, and find that 1,930 ChatGPT-generated code snippets suffer from maintainability issues. Subsequently, we investigate ChatGPT’s self-repairing ability and its interaction with static analysis tools to fix the errors uncovered in the previous step. Experiments suggest that ChatGPT can partially address these challenges, improving code quality by more than 20%, but there are still limitations and opportunities for improvement. Overall, our study provides valuable insights into the current limitations of ChatGPT and offers a roadmap for future research and development efforts to enhance the code generation capabilities of AI models like ChatGPT.
Tue 24 JunDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
10:30 - 12:30 | Code Generation 2Research Papers / Journal First at Cosmos Hall Chair(s): Reyhaneh Jabbarvand University of Illinois at Urbana-Champaign | ||
10:30 20mTalk | An Empirical Study of the Non-determinism of ChatGPT in Code Generation Journal First Shuyin Ouyang King's College London, Jie M. Zhang King's College London, Mark Harman Meta Platforms, Inc. and UCL, Meng Wang University of Bristol | ||
10:50 20mTalk | Don’t Complete It! Preventing Unhelpful Code Completion for Productive and Sustainable Neural Code Completion Systems Journal First Zhensu Sun Singapore Management University, Xiaoning Du Monash University, Fu Song Institute of Software at Chinese Academy of Sciences; University of Chinese Academy of Sciences; Nanjing Institute of Software Technology, Shangwen Wang National University of Defense Technology, Mingze Ni University of Technology Sydney, Li Li Beihang University, David Lo Singapore Management University | ||
11:10 20mTalk | Divide-and-Conquer: Generating UI Code from Screenshots Research Papers Yuxuan Wan The Chinese University of Hong Kong, Chaozheng Wang The Chinese University of Hong Kong, Yi Dong The Chinese University of Hong Kong, Wenxuan Wang Chinese University of Hong Kong, Shuqing Li The Chinese University of Hong Kong, Yintong Huo Singapore Management University, Michael Lyu Chinese University of Hong Kong DOI | ||
11:30 20mTalk | LLM-based Method Name Suggestion with Automatically Generated Context-Rich Prompts Research Papers Waseem Akram Beijing Institute of Technology, Yanjie Jiang Peking University, Yuxia Zhang Beijing Institute of Technology, Haris Ali Khan Beijing Institute of Technology, Hui Liu Beijing Institute of Technology DOI | ||
11:50 20mTalk | Beyond Functional Correctness: Investigating Coding Style Inconsistencies in Large Language Models Research Papers Yanlin Wang Sun Yat-sen University, Tianyue Jiang Sun Yat-sen University, Mingwei Liu Sun Yat-Sen University, Jiachi Chen Sun Yat-sen University, Mingzhi Mao Sun Yat-sen University, Xilin Liu Huawei Cloud, Yuchi Ma Huawei Cloud Computing Technologies, Zibin Zheng Sun Yat-sen University DOI | ||
12:10 20mTalk | Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues Journal First Yue Liu Monash University, Le-Cong Thanh The University of Melbourne, Ratnadira Widyasari Singapore Management University, Singapore, Kla Tantithamthavorn Monash University, Li Li Beihang University, Xuan-Bach D. Le University of Melbourne, David Lo Singapore Management University | ||
This is the main event hall of Clarion Hotel, which will be used to host keynote talks and other plenary sessions. The FSE and ISSTA banquets will also happen in this room.
The room is just in front of the registration desk, on the other side of the main conference area. The large doors with numbers “1” and “2” provide access to the Cosmos Hall.