Compiler construction is one of the oldest areas of software engineering, but despite its maturity it does have some underdeveloped sides. One of such sides is compiler testing: there have been many disparate methods developed for testing parsers, optimisers and other components, but no unified methodology that can be consumed by practitioners from a book and applied directly to their needs.

Instead of striving for covering all theoretical aspects of compiler testing in one paper, we present a case study. For an ongoing project of a relative large size for our company (two years, 3–6 devs, 510kLOC as of now), which is a clean room compiler development effort in replicating a 4GL from scratch, we had to build not only a testing framework, but a model-based test data generator. This tool consumes manually written specifications and generates all the necessary test code in the software language being developed, in the host language, and in auxiliary DSLs (batch files, XML project descriptions), to be able to satisfy both the developers and the customer. The number of specifications is 818 at the time of submission, while the number of test cases generated from them, is 5233. All these tests have been run prior to shipping for the last 32 releases of the compiler, both to ensure the lack of regression and to report on the project overall progress. The generated tests are separated into 11 categories which the paper details in the hope that the classification will aid in seeking related work and in pushing this line of research forward.