Fuzzing is an automated software testing technique used to find software vulnerabilities that works by sending large amounts of inputs to a software system to trigger bad behaviors. In recent years, the open-source software ecosystem has seen a significant increase in the adoption of fuzzing to avoid spreading vulnerabilities throughout the ecosystem. While fuzzing is the state-of-the-art approach to uncover vulnerabilities, there is currently a lack of knowledge regarding the challenges of conducting and maintaining fuzzing activities over time. Specifically, fuzzers are very complex tools to set up and build before they can be used. Quantitative and Qualitative analyses. We set out to empirically find out how challenging is build maintenance in the context of fuzzing. We mine over 1.2 million build logs from Google’s OSS-Fuzz service to investigate fuzzing build failures. We first conduct a quantitative analysis to quantify the prevalence of fuzzing build failures. Our results show that most open-source projects carefully manage their fuzzing builds and fix fuzzing build failures very quickly. We then manually investigate 677 failing fuzzing builds logs and establish a taxonomy of 25 root causes of build failures. Our taxonomy can serve as a reference for practitioners conducting fuzzing build maintenance.

Automated build failure classification. From our labeled dataset, we then extract and highlight common patterns of failures from the build logs. Using these patterns and our 677 labeled build logs, we finally train a machine learning model to recognize common failure patterns in failing build logs. Our model was able to achieve an F1-score of over 85% for three of the five most common types of failures when trying to automatically classify the root cause of failure in failing builds. Our modeling experiment shows the potential of using automation to simplify the process of fuzzing.