Wed 11 May 2022 05:10 - 05:15 at ICSE room 2-odd hours - Program Comprehension 1 Chair(s): Prajish Prasad
Source code summaries are important for the comprehension and maintenance of programs. However, there are plenty of programs with missing, outdated, or mismatched summaries. Recently, deep learning techniques have been exploited to automatically generate summaries for given code snippets. To achieve a profound understanding of how far we are from solving this problem and provide suggestions to future research, in this paper, we conduct a systematic and in-depth analysis of 5 state-of-the-art neural code summarization models on 6 widely used BLEU variants, 4 pre-processing operations and their combinations, and 3 widely used datasets. The evaluation results show that some important factors have a large impact on the model evaluation especially the performance of the model and the ranking among the models. However, these factors might be easily overlooked, which leads us not clearly knowing where we are now. Specially, (1) the BLEU metric widely used in existing work of evaluating code summarization models has many variants. Ignoring the difference among these variants could greatly affect the validity of the claimed results. Furthermore, we discover and solve the important and previously unknown bug about BLEU calculation in a commonly-used software package. Besides, we conduct human evaluation and find that metric BLEU-DC is most correlated to human perception; (2) code pre-processing choices can have a large (from -18% to +25%) impact on the summarization performance and should not be neglected. We also explore the aggregation of pre-processing combinations and boost the performance of models;(3) some important characteristics of datasets (corpus size, data splitting method, and duplication ratio) have a significant impact on model evaluation. Based on the experimental results, we build a shared code summarization toolbox to serve future research and give actionable suggestions on more systematic ways for evaluating code summarization and choosing the best method in different scenarios. We also suggest possible future research directions.
Tue 10 MayDisplayed time zone: Eastern Time (US & Canada) change
13:00 - 14:00 | Program Comprehension 5Journal-First Papers / Technical Track at ICSE room 1-odd hours Chair(s): Fabio Petrillo École de technologie supérieure (ÉTS), Montréal -- Université du Québec | ||
13:00 5mTalk | Journal First Submission of the Article: What do class comments tell us? An investigation of comment evolution and practices in Pharo Smalltalk Journal-First Papers Pooja Rani University of bern, Sebastiano Panichella Zurich University of Applied Sciences, Manuel Leuenberger Software Composition Group, University of Bern, Switzerland, Mohammad Ghafari School of Computer Science, University of Auckland, Oscar Nierstrasz University of Bern, Switzerland Link to publication DOI Authorizer link Media Attached | ||
13:05 5mTalk | Retrieving Data Constraint Implementations Using Fine-Grained Code Patterns Technical Track Juan Manuel Florez The University of Texas at Dallas, Jonathan Perry The University of Texas at Dallas, Shiyi Wei University of Texas at Dallas, Andrian Marcus University of Texas at Dallas Pre-print Media Attached | ||
13:10 5mTalk | On the Evaluation of Neural Code Summarization Technical Track Ensheng Shi Xi'an Jiaotong University, Yanlin Wang Microsoft Research, Lun Du Microsoft Research Asia, Junjie Chen Tianjin University, Shi Han Microsoft Research, Hongyu Zhang University of Newcastle, Dongmei Zhang Microsoft Research, Hongbin Sun Xi'an Jiaotong University DOI Pre-print Media Attached | ||
13:15 5mTalk | FIRA: Fine-Grained Graph-Based Code Change Representation for Automated Commit Message Generation Technical Track Jinhao Dong Peking University, Yiling Lou Purdue University, Qihao Zhu Peking University, Zeyu Sun Peking University, Zhilin Li Peking University, Wenjie Zhang Peking University, Dan Hao Peking University Pre-print Media Attached | ||
Wed 11 MayDisplayed time zone: Eastern Time (US & Canada) change
05:00 - 06:00 | Program Comprehension 1Technical Track / NIER - New Ideas and Emerging Results at ICSE room 2-odd hours Chair(s): Prajish Prasad IIT Bombay | ||
05:00 5mTalk | Supporting program comprehension by generating abstract code summary tree NIER - New Ideas and Emerging Results Avijit Bhattacharjee University of Saskatchewan, Canada, Banani Roy University of Saskatchewan, Kevin Schneider University of Saskatchewan DOI Pre-print Media Attached | ||
05:05 5mTalk | Practitioners’ Expectations on Automated Code Comment Generation Technical Track Xing Hu Zhejiang University, Xin Xia Huawei Software Engineering Application Technology Lab, David Lo Singapore Management University, Zhiyuan Wan Zhejiang University, Qiuyuan Chen Zhejiang University, Thomas Zimmermann Microsoft Research DOI Pre-print Media Attached | ||
05:10 5mTalk | On the Evaluation of Neural Code Summarization Technical Track Ensheng Shi Xi'an Jiaotong University, Yanlin Wang Microsoft Research, Lun Du Microsoft Research Asia, Junjie Chen Tianjin University, Shi Han Microsoft Research, Hongyu Zhang University of Newcastle, Dongmei Zhang Microsoft Research, Hongbin Sun Xi'an Jiaotong University DOI Pre-print Media Attached | ||
05:15 5mTalk | Bridging Pre-trained Models and Downstream Tasks for Source Code Understanding Technical Track Deze Wang National University of Defense Technology, Zhouyang Jia National University of Defense Technology, Shanshan Li National University of Defense Technology, Yue Yu College of Computer, National University of Defense Technology, Changsha 410073, China, Yun Xiong Fudan University, Wei Dong School of Computer, National University of Defense Technology, China, Liao Xiangke National University of Defense Technology Pre-print Media Attached | ||
05:20 5mTalk | FIRA: Fine-Grained Graph-Based Code Change Representation for Automated Commit Message Generation Technical Track Jinhao Dong Peking University, Yiling Lou Purdue University, Qihao Zhu Peking University, Zeyu Sun Peking University, Zhilin Li Peking University, Wenjie Zhang Peking University, Dan Hao Peking University Pre-print Media Attached | ||