Generation-based Code Review Automation: How Far Are We?
Code review is an effective software quality assurance activity; however, it is labor-intensive and time-consuming. Thus, a number of generation-based automatic code review (ACR) approaches have been proposed recently, which leverage deep learning techniques to automate various activities in the code review process (e.g., code revision generation and review comment generation).
We find the previous works carry three main limitations. First, the ACR approaches have been shown to be beneficial in each work, but those methods are not comprehensively compared with each other to show their superiority over their peer ACR approaches. Second, general-purpose pre-trained models such as CodeT5 are proven to be effective in a wide range of Software Engineering (SE) tasks. However, no prior work has investigated the effectiveness of these models in ACR tasks yet. Third, prior works heavily rely on the Exact Match (EM) metric which only focuses on the perfect predictions and ignores the positive progress made by incomplete answers. To fill such a research gap, we conduct a comprehensive study by comparing the effectiveness of recent ACR tools as well as the general-purpose pre-trained models. The results show that a general-purpose pre-trained model CodeT5 can outperform other models in most cases. Specifically, CodeT5 outperforms the prior state-of-the-art by 13.4%–38.9% in two code revision generation tasks. In addition, we introduce a new metric namely Edit Progress (EP) to quantify the partial progress made by ACR tools. The results show that the rankings of models for each task could be changed according to whether EM or EP is being utilized. Lastly, we derive several insightful lessons from the experimental results and reveal future research directions for generation-based code review automation.
Tue 16 MayDisplayed time zone: Hobart change
11:00 - 12:30
|REMS: Recommending Extract Method Refactoring Opportunities via Multi-view Representation of Code Property Graph
|Automating Method Naming with Context-Aware Prompt-Tuning
Jie Zhu Institute of Software, Chinese Academy of Sciences;University of Chinese Academy of Sciences, Lingwei Li Institute of Software at Chinese Academy of Sciences; University of Chinese Academy of Sciences, Li Yang Institute of Software at Chinese Academy of Sciences, Xiaoxiao Ma Institute of Software, Chinese Academy of Sciences, Chun Zuo SinosoftPre-print
|Generation-based Code Review Automation: How Far Are We?
Xin Zhou Singapore Management University, Singapore, Kisub Kim Singapore Management University, Bowen Xu North Carolina State University, DongGyun Han Royal Holloway, University of London, Junda He Singapore Management University, David Lo Singapore Management UniversityPre-print
|Reanalysis of Empirical Data on Java Local Variables with Narrow and Broad Scope
Dror Feitelson Hebrew UniversityPre-print
|Predicting vulnerability inducing function versions using node embeddings and graph neural networks
|Properly Offer Options to Improve the Practicality of Software Document Completion Tools
Early Research Achievements (ERA)
Zhipeng Cai School of Computer Science, Wuhan University, Songqiang Chen School of Computer Science, Wuhan University, Xiaoyuan Xie School of Computer Science, Wuhan University, ChinaMedia Attached