Software testing is difficult, tedious, and costs an estimated $48–87 billion USD/year in US labor. Automatic test generation tools aim to ease this burden but have important trade-offs. Fuzzers use an implicit oracle that can detect obviously invalid results. However, there is no general solution to the oracle problem, and an implicit oracle cannot automatically evaluate correctness. Test suite generators like EvoSuite use the program under test as the oracle and therefore cannot evaluate correctness. Property-based testing (PBT) tools evaluate correctness, but users find it difficult to come up with properties to test, and to understand whether the properties are correct. Consequently, adoption of these tools has been narrow, and test suites continue to be created manually by practitioners, who often use an example-based oracle to specify correct input and output examples.

To help bridge the gaps among oracles and tools, we present a Composite Oracle that incorporates implicit, property-based, and example-based oracles. To help us understand the practical properties of a Composite Oracle, we built TerzoN, an Automatic Test sUite Generator (ATUG) that implements a Composite Oracle. TerzoN displays all the test results in an integrated view composed from the results of the 3 types of oracles and finds some types of test assertion inconsistencies that might otherwise lead to misleading test results. We evaluated TerzoN with its Composite Oracle in a randomized controlled human trial with 14 professional software engineers using a popular industry tool, fast-check, as the control. Participants using TerzoN elicited 72% more bugs (p<0.01), accurately described more than twice the number of bugs (p<0.01) and tested 16% more quickly (p<0.05).