Combinatorial testing (CT) is an effective test design technique, considered to be a testing best practice. CT provides automatic test plan generation but requires a manual definition of the test space in the form of a combinatorial model. The theory of CT exists since the mid-80s, however when we started applying it to IBM products over a decade ago, we encountered several challenges that limited its applicability in practice. In this talk I will describe different real-world challenges in the application of CT in practice, and our solutions for overcoming these challenges. The challenges include scalability and verifiability of modeling large test spaces with dozens of parameters and constraints, and requirements posed by practitioners to (1) consider manually written tests in the CT test generation process, (2) effectively update models and reuse test plans as the software under test evolves, and (3) reuse models across applications for components with similar business logic. Solving these challenges led to wide use of CT across IBM testing services.

Speaker bio: Rachel Tzoref-Brill is a Research Staff Member at IBM Research in Haifa. She received her Ph.D. in computer science from Tel Aviv University, Israel, and her M.Sc and B.Sc degrees from Technion, Israel Institute of Technology. She won an ACM SIGSOFT Outstanding Doctoral Dissertation Award. Her research interests include combinatorial testing, cloud quality and observability, and software test generation. She received an IBM Corporate Award for combinatorial testing innovations. She authored a book chapter on advances in combinatorial testing in the book series ‘Advances in Computers’, 2019. She served on the program committees of ESEC/FSE 2020-2022 and ICSE 2021 and 2023.