Static bug finders (also known as static code analyzers, e.g., FindBugs, SonarQube) have been widely-adopted by developers to find bugs in real-world software projects. They leverage predefined heuristic static analysis rules to scan source code or binary code of a software project, and report violations to these rules as warnings to be verified. However, the advantages of static bug finders are overshadowed by such issues as uncovered obvious bugs, false positives, etc. To improve these tools, many techniques have been proposed to filter out false positives reported or design new static analysis rules. Nevertheless, the under-performance of bug finders can also be caused by the incorrectness of current rules contained in the static bug finders, which is not explored yet. In this work, we propose a differential testing approach to detect bugs in the rules of four widely-used static bug finders, i.e., SonarQube, PMD, SpotBugs, and ErrorProne, and conduct a qualitative study about the bugs found. The experiment on 2,728 open source projects reveals 46 bugs in the static bug finders, among which 24 are fixed or confirmed and the left are awaiting confirmation. We also summarize 13 bug patterns in the static analysis rules based on their context and root causes, which can serve as the checklist for designing and implementing other rules and/or in other tools. This study indicates that the commonly-used static bug finders are not as reliable as they might have been envisaged. It not only demonstrates the effectiveness of our approach, but also highlights the need to continue improving the reliability of the static bug finders.
Tue 17 MayDisplayed time zone: Eastern Time (US & Canada) change
02:00 - 02:50 | Session 11: Debugging 2Research / Early Research Achievements (ERA) / Tool Demonstration at ICPC room Chair(s): Fernanda Madeiral KTH Royal Institute of Technology | ||
02:00 7mTalk | Context-based Cluster Fault Localization Research Junji Yu Chongqing University, Yan Lei School of Big Data & Software Engineering, Chongqing University, Huan Xie Chongqing University, Lingfeng Fu Chongqing University, Chunyan Liu Chongqing University Pre-print Media Attached | ||
02:07 4mTalk | A Study of Single Statement Bugs Involving Dynamic Language Features Early Research Achievements (ERA) Li Sui Massey University, New Zealand, Shawn Rasheed Massey University, Amjed Tahir Massey University, Jens Dietrich Victoria University of Wellington Pre-print Media Attached | ||
02:11 7mTalk | XAI4FL: Enhancing Spectrum-Based Fault Localization with Explainable Artificial Intelligence Research Ratnadira Widyasari Singapore Management University, Singapore, Gede Artha Azriadi Prana Singapore Management University, Stefanus Agus Haryono Singapore Management University, Yuan Tian Queens University, Kingston, Canada, Hafil Noer Zachiary Singapore Management University, David Lo Singapore Management University Pre-print | ||
02:18 4mTalk | Do Visual Issue Reports Help Developers Fix Bugs? – A Preliminary Study of Using Videos and Images to Report Issues on GitHub – Early Research Achievements (ERA) Hiroki Kuramoto Kyushu University, Masanari Kondo Kyushu University, Yutaro Kashiwa Kyushu University, Yuta Ishimoto Kyushu University, Kaze Shindo Kyushu University, Yasutaka Kamei Kyushu University, Naoyasu Ubayashi Kyushu University Media Attached | ||
02:22 7mTalk | Find Bugs in Static Bug Finders Research Junjie Wang Institute of Software at Chinese Academy of Sciences, Yuchao Huang Institute of Software Chinese Academy of Sciences, Song Wang York University, Qing Wang Institute of Software at Chinese Academy of Sciences Pre-print Media Attached | ||
02:29 4mTalk | didiffff: A Viewer for Comparing Changes in both Code and Execution Trace Tool Demonstration Tetsuya Kanda Osaka University, Kazumasa Shimari Nara Institute of Science and Technology, Katsuro Inoue Osaka University Pre-print Media Attached | ||
02:33 17mLive Q&A | Q&A-Paper Session 11 Research | ||