This paper presents the design and the constitution of BugOss, a real-world regression bug benchmark for empirical study of regression fuzzing techniques. To reproduce the actual project context where a regression bug was introduced, each bug case of BugOss pinpoints the exact bug-inducing commit and provides a specific test oracle considering the presence of other co-existing bugs. These bug cases are systematically extracted based on the information from the OSS-Fuzz issue tracker and the target project repositories while avoiding uncertainty and carefully managing potential threats of experiment validities.
BugOss currently comprises 20 real-world bug cases from 20 open-source C/C++ projects, which had been reported by the OSS-Fuzz projects and confirmed by the project maintainers. The empirical investigation with two baseline fuzzers (libfuzzer and AFL++) and two regression fuzzing techniques (AFLChurn and Change-aware Seed Selection) show that, with the bug cases in BugOss, the regression fuzzing techniques perform differently depending on the given project context. In addition, the experiments imply that BugOss encompasses various cases of regression bugs in real-world, thus the bug cases would be useful for empirically investigating regression fuzzing techniques.
We believe that BugOss offers researchers a useful basis of empirical investigation of regression fuzzing techniques. To the best of the authors’ knowledge, BugOss is the first benchmark with actual bug-inducing commits of real-world C/C++ projects under continuous fuzzing for empirical study of regression fuzzing techniques. All artifacts of the benchmark and the experiment results are publicly open for future research at the following URL: https://github.com/sdevlab/BugOss.

• Preprint: https://sdevlab.github.io/bugoss.pdf