It is very difficult to thoroughly test a compiler, and as a
consequence it is common for released versions of production compilers
to contain bugs that cause them to crash and to emit incorrect object
code.
We created alive-mutate, a mutation-based fuzzing tool that takes test
cases written by humans and randomly modifies them, based on the
hypothesis that while compiler developers are fundamentally good at
writing tests, they also tend to miss corner cases.
Alive-mutate is integrated with the Alive2 translation validation tool
for LLVM, which is useful because it checks the behavior of
optimizations for all possible values of input variables.
Alive-mutate is also integrated with the LLVM middle-end, allowing it
to perform mutations, optimizations, and formal verification of the
optimizations all within a single program—avoiding numerous sources
of overhead.
Alive-mutate's fuzzing throughput is 12x higher, on average, than a
fuzzing workflow that runs mutation, optimization, and formal
verification in separate processes.
So far we have used alive-mutate to find and report 33 previously
unknown bugs in LLVM.