ICSME 2024
Sun 6 - Fri 11 October 2024

This paper presents our findings on the automatic summarization of Java methods within Ericsson, a global telecommunications company. We evaluate the performance of an approach called Automatic Semantic Augmentation of Prompts (ASAP), which uses a Large Language Model (LLM) to generate leading summary comments for Java methods. ASAP enhances the LLM’s prompt context by integrating static program analysis and information retrieval techniques to identify similar exemplar methods along with their developer-written Javadocs, and serves as the baseline in our study. In contrast, we explore and compare the performance of four simpler approaches that do not require static program analysis, information retrieval, or the presence of exemplars as in the ASAP method. Our methods rely solely on the Java method body as input, making them lightweight and more suitable for rapid deployment in commercial software development environments. We conducted experiments on an Ericsson software project and replicated the study using two widely-used open-source Java projects, Guava and Elasticsearch, to ensure the reliability of our results. Performance was measured across eight metrics that capture various aspects of similarity. Notably, one of our simpler approaches performed as well as or better than the ASAP method on both the Ericsson project and the open-source projects. Additionally, we performed an ablation study to examine the impact of method names on Javadoc summary generation across our four proposed approaches and the ASAP method. By masking the method names and observing the generated summaries, we found that our approaches were statistically significantly less influenced by the absence of method names compared to the baseline. This suggests that our methods are more robust to variations in method names and may derive summaries more comprehensively from the method body than the ASAP approach.

Wed 9 Oct

Displayed time zone: Arizona change

13:30 - 15:00
Session 3: Code Completion, Generation, and SummarizationResearch Track / Industry Track at Abineau
Chair(s): Gabriele Bavota Software Institute @ Università della Svizzera Italiana
13:30
15m
Deep Learning-based Code Completion: On the Impact on Performance of Contextual InformationResearch Track Paper
Research Track
Matteo Ciniselli Università della Svizzera Italiana, Luca Pascarella ETH Zurich, Gabriele Bavota Software Institute @ Università della Svizzera Italiana
13:45
15m
On the Generalizability of Transformer Models to Code Completions of Different LengthsResearch Track Paper
Research Track
Nathan Cooper , Rosalia Tufano Università della Svizzera Italiana, Gabriele Bavota Software Institute @ Università della Svizzera Italiana, Denys Poshyvanyk William & Mary
14:00
15m
Can Developers Prompt? A Controlled Experiment for Code Documentation GenerationOpen Research ObjectResearch Object ReviewedResearch Track Paper
Research Track
Hans-Alexander Kruse Universität Hamburg, Tim Puhlfürß Universität Hamburg, Walid Maalej University of Hamburg
14:15
15m
Icing on the Cake: Automatic Code Summarization at EricssonIndustry Track Paper
Industry Track
Giriprasad Sridhara IBM Research Labs, Sujoy Roychowdhury Ericsson R&D, Sumit Soman Ericsson R&D, Ranjani H G Ericsson R&D, Ricardo Britto Ericsson / Blekinge Institute of Technology
Pre-print
14:30
10m
Precos: Project-specific Retrieval for Better Code SummarizationVideo presentationResearch Track Paper
Research Track
Tingwei Zhu Nanjing University, Zhong Li , Tian Zhang Nanjing University, Minxue Pan Nanjing University, Xuandong Li Nanjing University
14:40
10m
Improving Retrieval-Augmented Code Comment Generation by Retrieving for GenerationVideo presentationResearch Track Paper
Research Track
Lu Hanzhen , Zhongxin Liu Zhejiang University