Demystifying the Challenges and Benefits of Analyzing User-Reported Logs in Bug Reports
Thu 12 May 2022 12:05 - 12:10 at ICSE room 3-even hours - Software Testing 14 Chair(s): Brittany Johnson
Logs in bug reports provide important debugging information for developers. During the debugging process, developers need to study the bug report and examine user-provided logs to understand the system executions that lead to the problem. Intuitively, user-provided logs illustrate the problems that users encounter and may help developers with the debugging process. However, some logs may be incomplete or inaccurate, which can cause difficulty for developers to diagnose the bug, and thus, delay the bug fixing process. In this paper, we conduct an empirical study on the challenges that developers may encounter when analyzing the user-provided logs and their benefits. In particular, we study both log snippets and exception stack traces in bug reports. We conduct our study on 10 large-scale open-source systems with a total of 1,561 bug reports with logs (BRWL) and 7,287 bug reports without logs (BRNL). Our findings show that: 1) BRWL takes longer time (median ranges from 3 to 91 days) to resolve compared to BRNL (median ranges from 1 to 25 days). We also find that reporters may not attach accurate or sufficient logs (i.e., developers often ask for additional logs in the Comments section of a bug report), which extends the bug resolution time. 2) Logs often provide a good indication of where a bug is located. Most bug reports (73%) have overlaps between the classes that generate the logs and their corresponding fixed classes. However, there is still a large number of bug reports where there is no overlap between the logged and fixed classes. 3) Our manual study finds that there is often missing system execution information in the logs. Many logs only show the point of failure (e.g., exception) and do not provide a direct hint on the actual root cause. In fact, through call graph analysis, we find that 28% of the studied bug reports have the fixed classes reachable from the logged classes, while they are not visible in the logs attached in bug reports. In addition, some logging statements are removed in the source code as the system evolves, which may cause further challenges in analyzing the logs. In short, our findings highlight possible future research directions to better help practitioners attach or analyze logs in bug reports.
Tue 10 MayDisplayed time zone: Eastern Time (US & Canada) change
Thu 12 MayDisplayed time zone: Eastern Time (US & Canada) change
12:00 - 13:00 | Software Testing 14Technical Track / Journal-First Papers / SEIP - Software Engineering in Practice at ICSE room 3-even hours Chair(s): Brittany Johnson George Mason University | ||
12:00 5mTalk | To What Extent Do DNN-based Image Classification Models Make Unreliable Inferences? Journal-First Papers Yongqiang TIAN The Hong Kong University of Science and Technology; University of Waterloo, Shiqing Ma Rutgers University, Ming Wen Huazhong University of Science and Technology, Yepang Liu Southern University of Science and Technology, Shing-Chi Cheung Hong Kong University of Science and Technology, Xiangyu Zhang Purdue University DOI Pre-print Media Attached | ||
12:05 5mTalk | Demystifying the Challenges and Benefits of Analyzing User-Reported Logs in Bug Reports Journal-First Papers An Ran Chen Concordia University, Tse-Hsun (Peter) Chen Concordia University, Shaowei Wang University of Manitoba Link to publication Media Attached | ||
12:10 5mTalk | Surveying the Developer Experience of Flaky Tests SEIP - Software Engineering in Practice Owain Parry The University of Sheffield, Gregory Kapfhammer Allegheny College, Michael Hilton Carnegie Mellon University, USA, Phil McMinn University of Sheffield Pre-print Media Attached | ||
12:15 5mTalk | Fuzzing Class Specifications Technical Track Facundo Molina University of Rio Cuarto and CONICET, Argentina, Marcelo d'Amorim Federal University of Pernambuco, Nazareno Aguirre University of Rio Cuarto and CONICET, Argentina Pre-print Media Attached | ||
12:20 5mTalk | Demystifying the Dependency Challenge in Kernel Fuzzing Technical Track Yu Hao University of California at Riverside, USA, Hang Zhang Georgia Institute of Technology, Guoren Li UC Riverside, Xingyun Du UC Riverside, Zhiyun Qian University of California at Riverside, USA, Ardalan Amiri Sani UC Irvine Pre-print Media Attached | ||
12:25 5mTalk | Natural Attack for Pre-trained Models of Code Technical Track Zhou Yang Singapore Management University, Jieke Shi Singapore Management University, Junda He Singapore Management University, David Lo Singapore Management University DOI Pre-print Media Attached | ||