Which bugs are missed in code reviews: An empirical study on SmartSHARK dataset
In pull-based development systems, code reviews and pull request comments play important roles in improving code quality. In such systems, reviewers attempt to carefully check a piece of code by different unit tests. Unfortunately, sometimes they miss bugs in their review of pull requests, which lead to quality degradations of the systems. In other words, disastrous consequences occur when bugs are observed after merging the pull requests. The lack of a concrete understanding of these bugs led us to investigate them and categorize them. In this research, we try to identify missed bugs in pull requests of SmartSHARK dataset projects. Our contribution is twofold. First, we hypothesized merged pull requests that have code reviews, code review comments, or pull request comments after merging, may have missed bugs after the code review. We considered these merged pull requests as candidate pull requests having missed bugs. Based on our assumption, we obtained 3,261 candidate pull requests from 77 open-source GitHub projects. After two rounds of restrictive manual analysis, we found 187 bugs missed in 173 pull requests. In the first step, we found 224 buggy pull requests containing missed bugs after merging the pull requests. Secondly, we defined and finalized a taxonomy that is appropriate for the bugs that we found and then found the distribution of bug categories after analyzing those pull requests all over again. The categories of missed bugs in pull requests and their distributions are: semantic (51.34%), build (15.5%), analysis checks (9.09%), compatibility (7.49%), concurrency (4.28%), configuration (4.28%), GUI (2.14%), API (2.14%), security (2.14%), and memory (1.6%).
Wed 18 MayDisplayed time zone: Eastern Time (US & Canada) change
12:00 - 12:50 | Mining ChallengeMining Challenge / Technical Papers at MSR Main room - even hours Chair(s): Steffen Herbold TU Clausthal | ||
12:00 4mTalk | An Exploratory Study on Refactoring Documentation in Issues Handling Mining Challenge Eman Abdullah AlOmar Stevens Institute of Technology, Anthony Peruma Rochester Institute of Technology, Mohamed Wiem Mkaouer Rochester Institute of Technology, Christian D. Newman Rochester Institute of Technology, Ali Ouni ETS Montreal, University of Quebec Pre-print | ||
12:04 4mTalk | Between JIRA and GitHub: ASFBot and its Influence on Human Comments in Issue Trackers Mining Challenge Ambarish Moharil Eindhoven University of Technology, Dmitrii Orlov Eindhoven University of Technology, Samar Jameel Eindhoven University of Technology, Tristan Trouwen Eindhoven University of Technology, Nathan Cassee Eindhoven University of Technology, Alexander Serebrenik Eindhoven University of Technology Pre-print | ||
12:08 4mTalk | Is Refactoring Always a Good Egg? Exploring the Interconnection Between Bugs and Refactorings Mining Challenge File Attached | ||
12:12 4mTalk | On the Co-Occurrence of Refactoring of Test and Source Code Mining Challenge Pre-print Media Attached | ||
12:16 4mTalk | Refactoring Debt: Myth or Reality? An Exploratory Study on the Relationship Between Technical Debt and RefactoringBest Mining Challenge Paper Award Mining Challenge Anthony Peruma Rochester Institute of Technology, Eman Abdullah AlOmar Stevens Institute of Technology, Christian D. Newman Rochester Institute of Technology, Mohamed Wiem Mkaouer Rochester Institute of Technology, Ali Ouni ETS Montreal, University of Quebec Pre-print Media Attached | ||
12:20 4mTalk | Studying the Impact of Continuous Delivery Adoption on Bug-Fixing Time in Apache’s Open-Source Projects Mining Challenge Carlos Diego Andrade de Almeida Federal University of Ceará, Diego N. Feijó Federal University of Ceará, Lincoln Rocha Federal University of Ceará Media Attached | ||
12:24 4mTalk | Which bugs are missed in code reviews: An empirical study on SmartSHARK dataset Mining Challenge fatemeh khoshnoud Department of Computer Science and Engineering and IT; School of Electrical and Computer Engineering, Shiraz University, Ali Rezaei Nasab Department of Computer Science and Engineering and IT; School of Electrical and Computer Engineering, Shiraz University, Zahra Toudeji Department of Computer Science and Engineering and IT; School of Electrical and Computer Engineering, Shiraz University, Ashkan Sami Shiraz University | ||
12:28 22mLive Q&A | Discussions and Q&A Technical Papers | ||