Explaining Explanations: An Empirical Study of Explanations in Code Reviews
Code reviews are central for software quality assurance. Ideally, reviewers should explain their feedback to enable authors of code changes to understand the feedback and act accordingly. Different developers might need different explanations in different contexts. Therefore, assisting this process first requires understanding the types of explanations reviewers usually provide. The goal of this paper is to study the types of explanations used in code reviews and explore the potential of Large Language Models (LLMs), specifically ChatGPT, in generating these specific types. We extracted 793 code review comments from Gerrit and manually labeled them based on whether they contained a suggestion, an explanation, or both. Our analysis shows that 42% of comments only include suggestions without explanations. We categorized the explanations into seven distinct types including rule or principle, similar examples, and future implications. When measuring their prevalence, we observed that some explanations are used differently by novice and experienced reviewers. Our manual evaluation shows that, when the explanation type is specified, ChatGPT can correctly generate the explanation in 88 out of 90 cases. This foundational work highlights the potential for future automation in code reviews, which can assist developers in sharing and obtaining different types of explanations as needed, thereby reducing back-and-forth communication.
Tue 24 JunDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
10:30 - 12:20 | Code Review, Build, and ReleaseIdeas, Visions and Reflections / Industry Papers / Demonstrations / Research Papers / Journal First at Aurora A Chair(s): Peter Rigby Concordia University; Meta | ||
10:30 10mTalk | From Overload to Insight: Bridging Code Search and Code Review with LLMs Ideas, Visions and Reflections Nikitha Rao Carnegie Mellon University, Bogdan Vasilescu Carnegie Mellon University, Reid Holmes University of British Columbia | ||
10:40 20mTalk | Explaining Explanations: An Empirical Study of Explanations in Code Reviews Journal First Ratnadira Widyasari Singapore Management University, Singapore, Ting Zhang Singapore Management University, Abir Bouraffa University of Hamburg, Walid Maalej University of Hamburg, David Lo Singapore Management University | ||
11:00 10mTalk | Support, Not Automation: Towards AI-supported Code Review for Code Quality and Beyond Ideas, Visions and Reflections | ||
11:10 20mTalk | BitsAI-CR: Automated Code Review via LLM in Practice Industry Papers Tao Sun Beihang University, Jian Xu ByteDance, Yuanpeng Li ByteDance, Zhao Yan ByteDance, Ge Zhang ByteDance, Lintao Xie ByteDance, Lu Geng ByteDance, Zheng Wang University of Leeds, Yueyan Chen ByteDance, Qin Lin ByteDance, Wenbo Duan ByteDance, Kaixin Sui ByteDance, Yuanshuo Zhu ByteDance | ||
11:30 10mTalk | Visualising Developer Interactions in Code Reviews Demonstrations | ||
11:40 20mTalk | CXXCrafter: An LLM-Based Agent for Automated C/C++ Open Source Software Building Research Papers Zhengmin Yu Fudan University, Yuan Zhang Fudan University, Ming Wen Huazhong University of Science and Technology, Yinan Nie Fudan University, Zhang Wenhui Fudan University, Min Yang Fudan University DOI | ||
12:00 20mTalk | SmartNote: An LLM-Powered, Personalised Release Note Generator That Just Works Research Papers Farbod Daneshyan Peking University, Runzhi He Peking University, Jianyu Wu Peking University, Minghui Zhou Peking University DOI |
Aurora A is the first room in the Aurora wing.
When facing the main Cosmos Hall, access to the Aurora wing is on the right, close to the side entrance of the hotel.