ICPC 2024
Sun 14 - Sat 20 April 2024 Lisbon, Portugal
co-located with ICSE 2024

Existing studies on the use of Large Language Models (LLMs) in software development leverage methodologies that limit their scal- ability and require intensive manual data collection and analysis, for example, due to the use of video data or think-aloud protocols. We propose the use of a specialized tool capable of automatically collecting fine-grained, relevant data during experiments and case studies. It enables researchers to understand for example how often participants accept or reject suggestions made by LLMs and what kinds of prompts are more likely to trigger accepted suggestions, even in studies targeting a large number of participants. We imple- ment this idea as a Visual Studio Code plugin named AthenaLLM. It mimics the functionalities of GitHub Copilot and offers seamless integration with OpenAI API models like GPT-4 and GPT-3.5, and compatibility with other models providing an OpenAI-compatible API, e.g., Vicuña. It automatically collects data at a fine level of granularity and covers both the interactions of developers with their IDE and the products of such interactions. Thus, the proposed approach also reduces bias that the experimental process itself may introduce, e.g., due to the need for participants to verbalize their thoughts. In this paper we discuss the limitations of previous studies and how AthenaLLM could enable researchers to go both broader (in terms of number of participants) and deeper (in terms of the kinds of research questions that can be tackled).

Mon 15 Apr

Displayed time zone: Lisbon change

11:00 - 12:30
11:00
10m
Talk
Towards Summarizing Code Snippets Using Pre-Trained TransformersICPCICPC Full paper
Research Track
Antonio Mastropaolo Università della Svizzera italiana, Matteo Ciniselli Università della Svizzera Italiana, Luca Pascarella ETH Zurich, Rosalia Tufano Università della Svizzera Italiana, Emad Aghajani Software Institute, USI Università della Svizzera italiana, Gabriele Bavota Software Institute @ Università della Svizzera Italiana
Pre-print
11:10
10m
Talk
Generating Java Methods: An Empirical Assessment of Four AI-Based Code AssistantsICPCICPC Full paper
Research Track
Vincenzo Corso University of Milano - Bicocca, Leonardo Mariani University of Milano-Bicocca, Daniela Micucci University of Milano-Bicocca, Italy, Oliviero Riganelli University of Milano - Bicocca
Pre-print
11:20
10m
Talk
Analyzing Prompt Influence on Automated Method Generation: An Empirical Study with CopilotICPCICPC Full paper
Research Track
Ionut Daniel Fagadau University of Milano - Bicocca, Leonardo Mariani University of Milano-Bicocca, Daniela Micucci University of Milano-Bicocca, Italy, Oliviero Riganelli University of Milano - Bicocca
Pre-print
11:30
10m
Talk
Interpretable Online Log Analysis Using Large Language Models with Prompt StrategiesICPCICPC Full paper
Research Track
Yilun Liu Huawei co. LTD, Shimin Tao University of Science and Technology of China; Huawei co. LTD, Weibin Meng Huawei co. LTD, Jingyu Wang , Wenbing Ma Huawei co. LTD, Yuhang Chen University of Science and Technology of China, Yanqing Zhao Huawei co. LTD, Hao Yang Huawei co. LTD, Yanfei Jiang Huawei co. LTD
Pre-print
11:40
10m
Talk
Do Machines and Humans Focus on Similar Code? Exploring Explainability of Large Language Models in Code SummarizationICPCICPC RENE Paper
Replications and Negative Results (RENE)
Jiliang Li Vanderbilt University, Yifan Zhang Vanderbilt University, Zachary Karas Vanderbilt University, Collin McMillan University of Notre Dame, Kevin Leach Vanderbilt University, Yu Huang Vanderbilt University
Pre-print
11:50
10m
Talk
Knowledge-Aware Code Generation with Large Language ModelsICPCICPC Full paper
Research Track
Tao Huang Shandong Normal University, Zhihong Sun Shandong Normal University, Zhi Jin Peking University, Ge Li Peking University, Chen Lyu Shandong Normal University
Pre-print
12:00
8m
Talk
Enhancing Source Code Representations for Deep Learning with Static AnalysisICPCICPC ERA Paper
Early Research Achievements (ERA)
Xueting Guan University of Melbourne, Christoph Treude Singapore Management University
Pre-print
12:08
8m
Talk
AthenaLLM: Supporting Experiments with Large Language Models in Software DevelopmentICPCICPC Tools
Tool Demonstration
Benedito Fernando Albuquerque de Oliveira Federal University of Pernambuco, Fernando Castor University of Twente and Federal University of Pernambuco
12:16
14m
Talk
AI-Assisted Program Comprehension: Panel with SpeakersICPC
Discussion