We report on our experience at Google of deploying a variety of coverage-guided and black-box fuzzing techinques to identify functional, robustness and security-related bugs in a number of compilers for two graphics shading languages: the WebGPU Shading Language (WGSL) and Standard Portable Intermediate Representation (SPIR-V). We discuss our experience deploying grey-box fuzzing via libFuzzer on ClusterFuzz and OSS-Fuzz using (a) libFuzzer’s built-in mutators, (b) a number of custom mutators based on regular expression matching and principled program transforamtion, and (c) a custom mutator for WGSL that leverages existing mutation-based tooling for SPIR-V together with tools for translating SPIR-V into WGSL. We also describe our experience deploying several black box fuzzers on ClusterFuzz, including two that are based on new randomized program genetors for WGSL that we implemented as part of this work, which we have also used in a more targeted fashion for end-to-end testing of implementations of WebGPU from Google and Mozilla.

Throughout, we focus on issues that we believe may generalise to other deployments of fuzzing, including: tradeoffs between the engineering effort required to create and deploy each fuzzer and the nature and number of bugs they have uncovered; time wasted due to false alarms arising due to fuzzer misconfiguration or defects in the fuzzers themselves; cases where fuzzing has informed the design of the language associated with the compilers being fuzzed; and the difficulty of writing fuzzers that respect pragmatic assumptions made by the software under test.