Large Language Models are Few-Shot Summarizers: Multi-Intent Comment Generation via In-Context Learning
Code comment generation aims at generating natural language descriptions for a code snippet to facilitate developers’ program comprehension activities. Despite being studied for a long time, a bottleneck for existing approaches is that given a code snippet, they can only generate one comment while developers usually need to know information from diverse perspectives such as what is the functionality of this code snippet and how to use it. To tackle this limitation, this study empirically investigates the feasibility of utilizing large language models (LLMs) to generate comments that can fulfill developers’ diverse intents. Our intuition is based on the facts that (1) the code and its pairwise comment are used during the pre-training process of LLMs to build the semantic connection between the natural language and programming language, and (2) comments in the real-world projects, which are collected for the pre-training, usually contain different developers’ intents. We thus postulate that the LLMs can already understand the code from different perspectives after the pre-training. Indeed, experiments on two large-scale datasets demonstrate the rationale of our insights: by adopting the in-context learning paradigm and giving adequate prompts to the LLM (\eg providing it with ten or more examples), the LLM can significantly outperform a state-of-the-art supervised learning approach on generating comments with multiple intents. Results also show that customized strategies for constructing the prompts and post-processing strategies for reranking the results can both boost the LLM’s performances, which shed light on future research directions for using LLMs to achieve comment generation.
Wed 17 AprDisplayed time zone: Lisbon change
14:00 - 15:30 | Evolution 1Research Track / Journal-first Papers / Demonstrations / Industry Challenge Track at Amália Rodrigues Chair(s): Jonathan Sillito Brigham Young University | ||
14:00 15mTalk | Large Language Models are Few-Shot Summarizers: Multi-Intent Comment Generation via In-Context Learning Research Track Mingyang Geng National University of Defense Technology, Shangwen Wang National University of Defense Technology, Dezun Dong NUDT, Haotian Wang National University of Defense Technolog, Ge Li Peking University, Zhi Jin Peking University, Xiaoguang Mao National University of Defense Technology, Liao Xiangke National University of Defense Technology DOI Pre-print | ||
14:15 15mTalk | Block-based Programming for Two-Armed Robots: A Comparative Study Research Track Felipe Fronchetti Virginia Commonwealth University, Nico Ritschel University of British Columbia, Logan Schorr Virginia Commonwealth University, Chandler Barfield Virginia Commonwealth University, Gabriella Chang Virginia Commonwealth University, Rodrigo Spinola Virginia Commonwealth University, Reid Holmes University of British Columbia, David C. Shepherd Louisiana State University DOI Pre-print Media Attached | ||
14:30 15mTalk | Exploiting Library Vulnerability via Migration Based Automating Test Generation Research Track Zirui Chen , Xing Hu Zhejiang University, Xin Xia Huawei Technologies, Yi Gao Zhejiang University, Tongtong Xu Huawei, David Lo Singapore Management University, Xiaohu Yang Zhejiang University | ||
14:45 15mTalk | ReposVul: A Repository-Level High-Quality Vulnerability Dataset Industry Challenge Track Xinchen Wang Harbin Institute of Technology, Ruida Hu Harbin Institute of Technology, Shenzhen, Cuiyun Gao Harbin Institute of Technology, Xin-Cheng Wen Harbin Institute of Technology, Yujia Chen Harbin Institute of Technology, Shenzhen, Qing Liao Harbin Institute of Technology Pre-print File Attached | ||
15:00 7mTalk | JOG: Java JIT Peephole Optimizations and Tests from Patterns Demonstrations Zhiqiang Zang The University of Texas at Austin, Aditya Thimmaiah The University of Texas at Austin, Milos Gligoric The University of Texas at Austin DOI Pre-print | ||
15:07 7mTalk | Predicting the Change Impact of Resolving Defects by Leveraging the Topics of Issue Reports in Open Source Software Systems Journal-first Papers Maram Assi Queen's University, Safwat Hassan University of Toronto, Canada, Stefanos Georgiou Queen's University, Ying Zou Queen's University, Kingston, Ontario | ||
15:14 7mTalk | Assessing the Exposure of Software Changes Journal-first Papers Mehran Meidani University of Waterloo, Maxime Lamothe Polytechnique Montreal, Shane McIntosh University of Waterloo Link to publication Pre-print | ||
15:21 7mTalk | Responding to change over time: A longitudinal case study on changes in coordination mechanisms in large‑scale agile Journal-first Papers Marthe Berntzen University of Oslo, Viktoria Stray University of Oslo, Nils Brede Moe , Rashina Hoda Monash University |