Write a Blog >>
SLE 2020
Sun 15 - Fri 20 November 2020 Online Conference
co-located with SPLASH 2020
Mon 16 Nov 2020 13:20 - 13:40 at SPLASH - Chair(s): Vadim Zaytsev
Tue 17 Nov 2020 01:20 - 01:40 at SPLASH - Chair(s): Vadim Zaytsev

We report on our experience in using various grammar-based test suite generation methods to test 61 single-pass compilers that undergraduate students submitted for the practical project of a computer architecture course.

We show that (1) all test suites constructed systematically following different grammar coverage criteria fall far behind the instructor’s test suite in achieved code coverage, in the number of triggered semantic errors, and in detected failures and crashes; (2) a medium-sized positive random test suite triggers more crashes than the instructor’s test suite, but achieves lower code coverage and triggers fewer non-crashing errors; and (3) a combination of the systematic and random test suites performs as well or better than the instructor’s test suite in all aspects and identifies errors or crashes in every single submission.

We then develop a light-weight extension of the basic grammar-based testing framework to capture contextual constraints, by encoding scoping and typing information as ``semantic mark-up tokens'' in the grammar rules. These mark-up tokens are interpreted by a small generic core engine when the tests are printed, and tests with a syntactic structure that cannot be completed into a valid program by choosing appropriate identifiers are discarded. We formalize individual error models by overwriting individual mark-up tokens,and generate tests that are guaranteed to break specific contextual properties of the language. We show that a fully automatically generated random test suite with 15 error models achieves roughly the same coverage as the instructor’s test suite, and outperforms it in the number of detected failures and crashes; moreover, all failing tests indicate real errors, and we have detected errors in the instructor’s reference implementation.

Mon 16 Nov
Times are displayed in time zone: Central Time (US & Canada) change

13:00 - 14:20: SLE at SPLASH +12h
Chair(s): Vadim ZaytsevUniversity of Twente, The Netherlands
13:00 - 13:20
Talk
SLE
Christoff RossouwStellenbosch University, Bernd FischerStellenbosch University
13:20 - 13:40
Talk
SLE
Phillip van HeerdenStellenbosch University, Moeketsi RaselimoStellenbosch University, Konstantinos (Kostis) SagonasUppsala University, Sweden, Bernd FischerStellenbosch University
13:40 - 14:00
Talk
SLE
Chelsea BarraballStellenbosch University, Moeketsi RaselimoStellenbosch University, Bernd FischerStellenbosch University
14:00 - 14:20
Talk
SLE
Dimitri RacordonUniversity of Geneva, Centre Universitaire d'Informatique, Geneva, Switzerland, Didier BuchsUniversity of Geneva, Centre Universitaire d'Informatique, Geneva, Switzerland

Tue 17 Nov
Times are displayed in time zone: Central Time (US & Canada) change

01:00 - 02:20: SLE at SPLASH
Chair(s): Vadim ZaytsevUniversity of Twente, The Netherlands
01:00 - 01:20
Talk
SLE
Christoff RossouwStellenbosch University, Bernd FischerStellenbosch University
01:20 - 01:40
Talk
SLE
Phillip van HeerdenStellenbosch University, Moeketsi RaselimoStellenbosch University, Konstantinos (Kostis) SagonasUppsala University, Sweden, Bernd FischerStellenbosch University
01:40 - 02:00
Talk
SLE
Chelsea BarraballStellenbosch University, Moeketsi RaselimoStellenbosch University, Bernd FischerStellenbosch University
02:00 - 02:20
Talk
SLE
Dimitri RacordonUniversity of Geneva, Centre Universitaire d'Informatique, Geneva, Switzerland, Didier BuchsUniversity of Geneva, Centre Universitaire d'Informatique, Geneva, Switzerland