ICPC 2024
Sun 14 - Sat 20 April 2024 Lisbon, Portugal
co-located with ICSE 2024

AI-based code assistants are promising tools that can facilitate and speed-up code development. They exploit machine learning algorithms and natural language processing to interact with developers, suggesting code snippets (e.g., method implementations) that can be incorporated into projects. Recent studies empirically investigated the effectiveness of code assistants using simple exemplary problems (e.g., the re-implementation of well-known algorithms), which fail to capture the spectrum and nature of the tasks actually faced by developers. In this paper, we expand the knowledge in the area by comparatively assessing four popular AI-based code assistants, namely GitHub Copilot, Tabnine, ChatGPT, and Google Bard, with a dataset of 100 methods that we constructed from real-life open source Java projects, considering a variety of cases for complexity and dependency from contextual elements. Results show that Copilot is often more accurate than other techniques, yet none of the assistants is completely subsumed by the rest of the approaches. Interestingly, the effectiveness of these solutions dramatically decreases when dealing with dependencies outside the boundaries of single classes.

Mon 15 Apr

Displayed time zone: Lisbon change

11:00 - 12:30
11:00
10m
Talk
Towards Summarizing Code Snippets Using Pre-Trained TransformersICPCICPC Full paper
Research Track
Antonio Mastropaolo Università della Svizzera italiana, Matteo Ciniselli Università della Svizzera Italiana, Luca Pascarella ETH Zurich, Rosalia Tufano Università della Svizzera Italiana, Emad Aghajani Software Institute, USI Università della Svizzera italiana, Gabriele Bavota Software Institute @ Università della Svizzera Italiana
Pre-print
11:10
10m
Talk
Generating Java Methods: An Empirical Assessment of Four AI-Based Code AssistantsICPCICPC Full paper
Research Track
Vincenzo Corso University of Milano - Bicocca, Leonardo Mariani University of Milano-Bicocca, Daniela Micucci University of Milano-Bicocca, Italy, Oliviero Riganelli University of Milano - Bicocca
Pre-print
11:20
10m
Talk
Analyzing Prompt Influence on Automated Method Generation: An Empirical Study with CopilotICPCICPC Full paper
Research Track
Ionut Daniel Fagadau University of Milano - Bicocca, Leonardo Mariani University of Milano-Bicocca, Daniela Micucci University of Milano-Bicocca, Italy, Oliviero Riganelli University of Milano - Bicocca
Pre-print
11:30
10m
Talk
Interpretable Online Log Analysis Using Large Language Models with Prompt StrategiesICPCICPC Full paper
Research Track
Yilun Liu Huawei co. LTD, Shimin Tao University of Science and Technology of China; Huawei co. LTD, Weibin Meng Huawei co. LTD, Jingyu Wang , Wenbing Ma Huawei co. LTD, Yuhang Chen University of Science and Technology of China, Yanqing Zhao Huawei co. LTD, Hao Yang Huawei co. LTD, Yanfei Jiang Huawei co. LTD
Pre-print
11:40
10m
Talk
Do Machines and Humans Focus on Similar Code? Exploring Explainability of Large Language Models in Code SummarizationICPCICPC RENE Paper
Replications and Negative Results (RENE)
Jiliang Li Vanderbilt University, Yifan Zhang Vanderbilt University, Zachary Karas Vanderbilt University, Collin McMillan University of Notre Dame, Kevin Leach Vanderbilt University, Yu Huang Vanderbilt University
Pre-print
11:50
10m
Talk
Knowledge-Aware Code Generation with Large Language ModelsICPCICPC Full paper
Research Track
Tao Huang Shandong Normal University, Zhihong Sun Shandong Normal University, Zhi Jin Peking University, Ge Li Peking University, Chen Lyu Shandong Normal University
Pre-print
12:00
8m
Talk
Enhancing Source Code Representations for Deep Learning with Static AnalysisICPCICPC ERA Paper
Early Research Achievements (ERA)
Xueting Guan University of Melbourne, Christoph Treude Singapore Management University
Pre-print
12:08
8m
Talk
AthenaLLM: Supporting Experiments with Large Language Models in Software DevelopmentICPCICPC Tools
Tool Demonstration
Benedito Fernando Albuquerque de Oliveira Federal University of Pernambuco, Fernando Castor University of Twente and Federal University of Pernambuco
12:16
14m
Talk
AI-Assisted Program Comprehension: Panel with SpeakersICPC
Discussion