ICPC 2024
Sun 14 - Sat 20 April 2024 Lisbon, Portugal
co-located with ICSE 2024

Recent language models have demonstrated proficiency in summarizing source code. However, as in many other domains of machine learning, language models of code lack sufficient explainability – informally, we lack a formulaic or intuitive understanding of what and how models learn from code. Explainability of language models can be partially provided if, as the models learn to produce higher-quality code summaries, they also align in deeming the same code parts important as those identified by human programmers. In this paper, we report negative results from our investigation of explainability of language models in code summarization through the lens of human comprehension. We measure human focus on code using eye-tracking metrics such as fixation counts and duration in code summarization tasks. To approximate language model focus, we employ a state-of-the-art model-agnostic, black-box, perturbation-based approach, SHAP (SHapley Additive exPlanations), to identify which code tokens influence that generation of summaries. Using these settings, we find no statistically significant relationship between language models’ focus and human programmers’ attention. Furthermore, alignment between model and human foci in this setting does not seem to dictate the quality of the LLM-generated summaries. Our study highlights an inability to align human focus with SHAP-based model focus measures. This result calls for future investigation of multiple open questions for explainable language models for code summarization and software engineering tasks in general, including the training mechanisms of language models for code, whether there is an alignment between human and model attention on code, whether human attention can improve the development of language models, and what other model focus measures are appropriate for improving explainability.

Mon 15 Apr

Displayed time zone: Lisbon change

11:00 - 12:30
11:00
10m
Talk
Towards Summarizing Code Snippets Using Pre-Trained TransformersICPCICPC Full paper
Research Track
Antonio Mastropaolo Università della Svizzera italiana, Matteo Ciniselli Università della Svizzera Italiana, Luca Pascarella ETH Zurich, Rosalia Tufano Università della Svizzera Italiana, Emad Aghajani Software Institute, USI Università della Svizzera italiana, Gabriele Bavota Software Institute @ Università della Svizzera Italiana
Pre-print
11:10
10m
Talk
Generating Java Methods: An Empirical Assessment of Four AI-Based Code AssistantsICPCICPC Full paper
Research Track
Vincenzo Corso University of Milano - Bicocca, Leonardo Mariani University of Milano-Bicocca, Daniela Micucci University of Milano-Bicocca, Italy, Oliviero Riganelli University of Milano - Bicocca
Pre-print
11:20
10m
Talk
Analyzing Prompt Influence on Automated Method Generation: An Empirical Study with CopilotICPCICPC Full paper
Research Track
Ionut Daniel Fagadau University of Milano - Bicocca, Leonardo Mariani University of Milano-Bicocca, Daniela Micucci University of Milano-Bicocca, Italy, Oliviero Riganelli University of Milano - Bicocca
Pre-print
11:30
10m
Talk
Interpretable Online Log Analysis Using Large Language Models with Prompt StrategiesICPCICPC Full paper
Research Track
Yilun Liu Huawei co. LTD, Shimin Tao University of Science and Technology of China; Huawei co. LTD, Weibin Meng Huawei co. LTD, Jingyu Wang , Wenbing Ma Huawei co. LTD, Yuhang Chen University of Science and Technology of China, Yanqing Zhao Huawei co. LTD, Hao Yang Huawei co. LTD, Yanfei Jiang Huawei co. LTD
Pre-print
11:40
10m
Talk
Do Machines and Humans Focus on Similar Code? Exploring Explainability of Large Language Models in Code SummarizationICPCICPC RENE Paper
Replications and Negative Results (RENE)
Jiliang Li Vanderbilt University, Yifan Zhang Vanderbilt University, Zachary Karas Vanderbilt University, Collin McMillan University of Notre Dame, Kevin Leach Vanderbilt University, Yu Huang Vanderbilt University
Pre-print
11:50
10m
Talk
Knowledge-Aware Code Generation with Large Language ModelsICPCICPC Full paper
Research Track
Tao Huang Shandong Normal University, Zhihong Sun Shandong Normal University, Zhi Jin Peking University, Ge Li Peking University, Chen Lyu Shandong Normal University
Pre-print
12:00
8m
Talk
Enhancing Source Code Representations for Deep Learning with Static AnalysisICPCICPC ERA Paper
Early Research Achievements (ERA)
Xueting Guan University of Melbourne, Christoph Treude Singapore Management University
Pre-print
12:08
8m
Talk
AthenaLLM: Supporting Experiments with Large Language Models in Software DevelopmentICPCICPC Tools
Tool Demonstration
Benedito Fernando Albuquerque de Oliveira Federal University of Pernambuco, Fernando Castor University of Twente and Federal University of Pernambuco
12:16
14m
Talk
AI-Assisted Program Comprehension: Panel with SpeakersICPC
Discussion

Hide past events