Generating valid test inputs for a program is much easier if one knows the input language. In this talk, I introduce and demonstrate techniques that given a program P without any input samples or models, learns an input grammar that represents the syntactically valid inputs for P—a grammar which can then be used for highly effective test generation for P. To this end, I introduce a test generator targeted at input parsers that systematically explores parsing alternatives based on dynamic tracking of constraints; the resulting inputs and associated equivalence classes are then fed to an extended grammar learner whose resulting grammar can then be used for producing test cases.

The combination of grammar learning and grammar-based test generation opens the door towards fully automatic and comprehensive program testing; within a few minutes, our Pygmalion prototype can infer grammars and generate thousands of inputs for programs where random approaches such as AFL or symbolic approaches such as KLEE would take hours to days for.

Joint work with Rahul Gopinath, Björn Mathis, Matthias Höschele, and Alexander Kampmann. Will do live demos. Source code available.

Andreas Zeller is Faculty at the Center for IT-Security, Privacy, and Accountability (CISPA), and professor for Software Engineering at Saarland University, both in Saarbrücken, Germany. His research concerns the analysis of large software systems and their development process. In 2010, Zeller was inducted as Fellow of the ACM for his contributions to automated debugging and mining software archives, for which he also was awarded 10-year impact awards from ACM SIGSOFT and ICSE. In 2011, he received an ERC Advanced Grant, Europe’s highest and most prestigious individual research grant, for work on specification mining and test case generation. In 2013, Zeller co-founded Testfabrik AG, a start-up on automatic testing of Web applications, where he chairs the supervisory board. In 2018, he received the highest research award of ACM SIGSOFT, the Outstanding Research Award.