CPerfSmith - A Randomized C Program Generator for Performance-Oriented Compiler Testing
Software testing is a critical stage in the software development lifecycle, ensuring that programs behave correctly and reliably across diverse environments. Unlike general software testing, compiler testing requires well-structured input programs that systematically exercise the compiler’s various optimization passes, since incomplete coverage can easily mask critical issues. Currently, a range of techniques exists for compiler testing, including differential testing, fuzzing, slicing, sanitizers and formal methods based validation tools. While these methods are effective at detecting compiler crashes, undefined behavior, or general correctness bugs, they are largely inadequate for systematically evaluating compiler optimizations.
Tools like csmith have pioneered random program generation for compiler testing, but they lack precise control over program structure, which is critical when the goal is to stress specific optimization passes or identify performance regressions. To address this limitation, we developed \textbf{CPerfSmith }, an extension of \textbf{CPerfSmith } that provides fine-grained control over code constructs, enabling targeted generation of programs that exercise particular compiler optimizations. Through our experiments, \textbf{CPerfSmith } was able to generate over 100 programs exhibiting significant runtime differences across industry-standard C compilers (GCC, Clang, and AOCC). Furthermore, we successfully created 30 test cases that revealed significant performance regressions between two subsequent release versions of Clang, demonstrating the tool’s ability to detect missing or improperly applied optimizations.