Write a Blog >>
ICSE 2022
Sun 8 - Fri 27 May 2022
Tue 10 May 2022 13:10 - 13:15 at ICSE room 1-odd hours - Program Comprehension 5 Chair(s): Fabio Petrillo
Wed 11 May 2022 05:10 - 05:15 at ICSE room 2-odd hours - Program Comprehension 1 Chair(s): Prajish Prasad

Source code summaries are important for the comprehension and maintenance of programs. However, there are plenty of programs with missing, outdated, or mismatched summaries. Recently, deep learning techniques have been exploited to automatically generate summaries for given code snippets. To achieve a profound understanding of how far we are from solving this problem and provide suggestions to future research, in this paper, we conduct a systematic and in-depth analysis of 5 state-of-the-art neural code summarization models on 6 widely used BLEU variants, 4 pre-processing operations and their combinations, and 3 widely used datasets. The evaluation results show that some important factors have a large impact on the model evaluation especially the performance of the model and the ranking among the models. However, these factors might be easily overlooked, which leads us not clearly knowing where we are now. Specially, (1) the BLEU metric widely used in existing work of evaluating code summarization models has many variants. Ignoring the difference among these variants could greatly affect the validity of the claimed results. Furthermore, we discover and solve the important and previously unknown bug about BLEU calculation in a commonly-used software package. Besides, we conduct human evaluation and find that metric BLEU-DC is most correlated to human perception; (2) code pre-processing choices can have a large (from -18% to +25%) impact on the summarization performance and should not be neglected. We also explore the aggregation of pre-processing combinations and boost the performance of models;(3) some important characteristics of datasets (corpus size, data splitting method, and duplication ratio) have a significant impact on model evaluation. Based on the experimental results, we build a shared code summarization toolbox to serve future research and give actionable suggestions on more systematic ways for evaluating code summarization and choosing the best method in different scenarios. We also suggest possible future research directions.

Tue 10 May

Displayed time zone: Eastern Time (US & Canada) change

13:00 - 14:00
Program Comprehension 5Journal-First Papers / Technical Track at ICSE room 1-odd hours
Chair(s): Fabio Petrillo École de technologie supérieure (ÉTS), Montréal -- Université du Québec
13:00
5m
Talk
Journal First Submission of the Article: What do class comments tell us? An investigation of comment evolution and practices in Pharo Smalltalk
Journal-First Papers
Pooja Rani University of bern, Sebastiano Panichella Zurich University of Applied Sciences, Manuel Leuenberger Software Composition Group, University of Bern, Switzerland, Mohammad Ghafari School of Computer Science, University of Auckland, Oscar Nierstrasz University of Bern, Switzerland
Link to publication DOI Authorizer link Media Attached
13:05
5m
Talk
Retrieving Data Constraint Implementations Using Fine-Grained Code Patterns
Technical Track
Juan Manuel Florez The University of Texas at Dallas, Jonathan Perry The University of Texas at Dallas, Shiyi Wei University of Texas at Dallas, Andrian Marcus University of Texas at Dallas
Pre-print Media Attached
13:10
5m
Talk
On the Evaluation of Neural Code Summarization
Technical Track
Ensheng Shi Xi'an Jiaotong University, Yanlin Wang Microsoft Research, Lun Du Microsoft Research Asia, Junjie Chen Tianjin University, Shi Han Microsoft Research, Hongyu Zhang University of Newcastle, Dongmei Zhang Microsoft Research, Hongbin Sun Xi'an Jiaotong University
DOI Pre-print Media Attached
13:15
5m
Talk
FIRA: Fine-Grained Graph-Based Code Change Representation for Automated Commit Message Generation
Technical Track
Jinhao Dong Peking University, Yiling Lou Purdue University, Qihao Zhu Peking University, Zeyu Sun Peking University, Zhilin Li Peking University, Wenjie Zhang Peking University, Dan Hao Peking University
Pre-print Media Attached

Wed 11 May

Displayed time zone: Eastern Time (US & Canada) change

05:00 - 06:00
05:00
5m
Talk
Supporting program comprehension by generating abstract code summary tree
NIER - New Ideas and Emerging Results
Avijit Bhattacharjee University of Saskatchewan, Canada, Banani Roy University of Saskatchewan, Kevin Schneider University of Saskatchewan
DOI Pre-print Media Attached
05:05
5m
Talk
Practitioners’ Expectations on Automated Code Comment Generation
Technical Track
Xing Hu Zhejiang University, Xin Xia Huawei Software Engineering Application Technology Lab, David Lo Singapore Management University, Zhiyuan Wan Zhejiang University, Qiuyuan Chen Zhejiang University, Thomas Zimmermann Microsoft Research
DOI Pre-print Media Attached
05:10
5m
Talk
On the Evaluation of Neural Code Summarization
Technical Track
Ensheng Shi Xi'an Jiaotong University, Yanlin Wang Microsoft Research, Lun Du Microsoft Research Asia, Junjie Chen Tianjin University, Shi Han Microsoft Research, Hongyu Zhang University of Newcastle, Dongmei Zhang Microsoft Research, Hongbin Sun Xi'an Jiaotong University
DOI Pre-print Media Attached
05:15
5m
Talk
Bridging Pre-trained Models and Downstream Tasks for Source Code Understanding
Technical Track
Deze Wang National University of Defense Technology, Zhouyang Jia National University of Defense Technology, Shanshan Li National University of Defense Technology, Yue Yu College of Computer, National University of Defense Technology, Changsha 410073, China, Yun Xiong Fudan University, Wei Dong School of Computer, National University of Defense Technology, China, Xiangke Liao National University of Defense Technology
Pre-print Media Attached
05:20
5m
Talk
FIRA: Fine-Grained Graph-Based Code Change Representation for Automated Commit Message Generation
Technical Track
Jinhao Dong Peking University, Yiling Lou Purdue University, Qihao Zhu Peking University, Zeyu Sun Peking University, Zhilin Li Peking University, Wenjie Zhang Peking University, Dan Hao Peking University
Pre-print Media Attached
Hide past events

Information for Participants
Tue 10 May 2022 13:00 - 14:00 at ICSE room 1-odd hours - Program Comprehension 5 Chair(s): Fabio Petrillo
Info for room ICSE room 1-odd hours:

Click here to go to the room on Midspace

Wed 11 May 2022 05:00 - 06:00 at ICSE room 2-odd hours - Program Comprehension 1 Chair(s): Prajish Prasad
Info for room ICSE room 2-odd hours:

Click here to go to the room on Midspace