IWCT 2024

UPDATE:

It is our great pleasure to announce that Dr. Ilias S. Kotsireas, Professor at Wilfrid Laurier University (Canada) [IK], will give a keynote at IWCT 2024!

His research interests are in the areas of symbolic computation and combinatorial designs, including theoretical and algorithmic aspects of solving systems of polynomial equations. Ilias has extensive expertise in algorithmic, meta-heuristic and high-performance computing techniques studying questions related to combinatorial designs, for example concerning existence. In his keynote, Ilias will cover topics of interest related to combinatorial testing, especially aiming towards the foundations of it and related optimization algorithms.

Ilias is Professor at Wilfrid Laurier University [IK], Director of the Computer Algebra Research Group of WLU [CARGOLAB] and a Fellow of the Institute of Combinatorics and its Applications. He is the Senior Editor of Maple Transactions, Editor-in-Chief of the Journal of Algebraic Combinatorics and holds multiple other senior editorial positions in journals. Ilias has published more than 200 contributions, in particular extensively in journals and conferences as well as serving on many occasions as Guest Editor for Special Issues of Journals and as Editor of Books.

[IK] https://web.wlu.ca/science/physcomp/ikotsireas/

[CARGOLAB] https://cargo.wlu.ca/

About IWCT

The International Workshop on Combinatorial Testing (IWCT) 2024 is to be held in conjunction with ICST 2024, focusing on combinatorial testing. The workshop welcomes academic research submissions, as well as industrial experience reports.

Combinatorial Testing (CT), or Combinatorial Interaction Testing (CIT), is a widely applicable generic methodology and technology for software verification and validation, considered a testing best practice. In a combinatorial test plan, all interactions between parameters up to a certain level are covered. For example, in pairwise testing, for every pair of parameters, every pair of values will appear at least once. Studies show that CT is more efficient and effective than random testing.

CT has gained significant interest in recent years, both in research and in practice. However, many issues still remain unresolved, and much research is still needed in the field. For example, while pairwise testing is a well recognized and popular test planning method, investigations of actual failures in a number of software and systems convincingly show that pairwise testing is usually not sufficient, so higher strength CT (i.e., t-way for t > 2) may be needed.

In addition, the combinatorial test plans need to exclude invalid combinations of test values that cannot be executed, which limits the degrees of freedom the algorithms have, thus complicating the problem. Moreover, modeling languages and tools for easily capturing the input test space are also required for real-life applicability of CT. Other obstacles for wide acceptance of CT in industry are the gap between the generated test plans and executable tests, and the difficulty in determining expected results for the generated tests. Finally, empirical studies on CT, as well as thorough comparison with other methods, are also required.

In this workshop, we plan to bring together researchers actively working on combinatorial testing, and create a productive and creative environment for sharing and collaboration. IWCT being the primary venue dedicated to CT, yet there are many researchers working in the field, we expect, like in previous years, to see high responsiveness to take part in the workshop. Researchers attending the workshop will have an opportunity to publish their work in a dedicated venue, create new collaborations and take active part in the growing community of researchers working in the field.

The workshop will also be a meeting place between academia and industry, thus uniting academic excellence and industrial experience and needs. This will allow participants from academia to learn about the industrial experience in practical application of CT to real-life testing problems, and together with the colleagues from industry identify the difficulties that are obstacle to wider application of CT, and should be addressed in future research. Industrial participants will have an opportunity to meet the leading scientists in the field, and hear about the latest advances and innovations.

Combinatorial test competition

IWCT 2024 will host the third edition of the competition among tools for combinatorial testing! Details are available here: https://fmselab.github.io/ct-competition/.

Update (Nov 22, 2023): Training benchmarks are now available!

Previous Editions of IWCT

We are proud of more than ten previous editions of IWCT, showing the maturity of the field and the support of our community:

12th IWCT 2023

11th IWCT 2022

10th IWCT 2021

9th IWCT 2020

8th IWCT 2019

7th IWCT 2018

6th IWCT 2017

5th IWCT 2016

4th IWCT 2015

3rd IWCT 2014

2nd IWCT 2013

1st IWCT 2012

Disclaimer: All text on the IWCT website has been carefully checked. Nevertheless, the organizers cannot guarantee the correctness, completeness and topicality of the information. Liability of the organizers is therefore excluded. The links to other websites have been carefully chosen. However, since the organizers have no say on their content, they assumes no responsibility for these.

Accepted Papers

Call for Papers

We invite submissions of high-quality papers presenting original work on both theoretical and experimental aspects of combinatorial testing (CT).

Each submitted paper must conform to the IEEE two-column conference publication format. All papers will be reviewed by at least three members from the program committee. If a submission is accepted, at least one author of the paper is required to attend the workshop in person with an author registration and present the paper for it to be published in the ICST 2024 workshop proceedings.

Submission

Details on how to submit will be announced soon.

Full and short papers

We invite submissions for full papers (up to 10 pages, including references) as well as short papers (up to 4 pages, including references). Accepted papers in these two categories – full and short – will be part of the proceedings of IWCT 2024.

Journal-First

The aim of the Journal-First (JF) submission category is to further enrich the program of IWCT, as well as to provide an overall more flexible path to publication and dissemination of original research that is within the scope of IWCT. A submission in this category must adhere to the following criteria:

1. It should be clearly within the scope of the workshop.
2. It should be recent: it should have been accepted and made publicly available in a journal (online or in print) by January 1, 2022 or more recently.
3. It has not been presented at, and is not under consideration for, Journal-First tracks of other conferences or workshops.
4. The submission has to be in the form of a 2-page extended abstract and has to provide a concise summary of the published journal paper.

Journal-First submissions must be marked as such in the submission’s title, and must explicitly include full bibliographic details (including a DOI) of the journal publication they are based on. Since the referenced journal papers have already been reviewed and accepted by the corresponding journals, submissions in the JF category will not be reviewed again for technical content. Submissions will be judged on the basis of the above criteria, but also considering how well they would complement the workshop’s technical program. Accepted submissions in this category will be part of the proceedings of IWCT 2024, with the title equal to the original title of the article with the prefix Summary of.

Posters

In addition to the traditional program topics from the past years, and taking advantage of the fact that some of the lead CT tool developers have expressed interest in participating again in the third iteration of the CT competition (see below), we would like to offer with the poster session a low-threshold opportunity – especially targeting industry – to present activities within the scope of the workshop to the participants of IWCT. A tool demonstration can be given along a poster. In case a poster contribution wishes to provide more details or information, but does not want to submit a short of full paper, the authors can choose to submit in addition to the poster a maximal 2-page extended abstract detailing the content of the poster. The title of the extended abstract has to have the prefix Extended Abstract of Poster and these extended abstracts – if accepted – will also appear in the proceedings of IWCT 2024.

CT competition

IWCT 2024 will also host the 3rd combinatorial testing competition for combinatorial test generation tools. Details can be found here.

Update (Nov 22, 2023): Training benchmarks are now available!

Special Issue in Section of Springer Nature Journal

We plan to organize a call among the best papers of the workshop to submit extended versions in the Springer Nature Computer Science journal in the Section Combinatorial Methods and Models in System Testing. More information will be provided at the workshop.

Topics for submissions

Topics of interest for submissions include, but are not limited to:

Combinatorial testing workflow

• Modeling the input space for CT – the input to CT algorithms consists of the desired strength of coverage, a set of parameters together with their respective values, and constraints on value combinations. This input should capture correctly the points of variability in testing the system. While this input is clearly crucial for the effectiveness of CT, it is a difficult problem.
• Efficient algorithms to generate test plans with small size for t-way testing for t > 2, involving support of constraints on combinations that are possible.
• Determination of expected system behavior for each test case – while the test cases are automatically generated by the CT algorithm, determining the expected system behavior is currently usually a manual task.
• Executing CT test plans – the result of CT algorithms is a list of tests, where a test is represented by a value assignment for each parameter. There is still significant effort in transforming this representation into actual tests that a tester or testing tool can execute.
• Combinatorial testing based fault localization.
• Implementation of CT in existing testing infrastructures.
• Handling changes in test requirements – current CT methods focus on one-time generation of a test plan from a given set of requirements, but test requirements are almost never static, and tend to change between different releases and versions. However, generating a new plan of tests for each release may not be practical either.

Real-world experience in deployment of combinatorial testing

• Empirical studies and feedback from practical applications of CT.
• CT targeting AI-based systems.
• CT targeting security, including hardware security.
• CT targeting concurrent and real-time systems.
• CT targeting cloud computing systems and use of combinatorial methods in cloud architecture.
• CT targeting feature models for software product lines.
• CT targeting other domains, e.g., study of gene regulations and other biotechnology applications as well as mechanical engineering.

Applicability of combinatorial testing

• Comparison and combination of CT with other dynamic verification methods.
• Investigation of historical records of failures to determine the kind of CT which may have detected underlying faults.
• Methodology used for test space modeling and determination of interaction coverage requirements.
• Evaluation and return of investment metrics to assess the degree of usefulness of CT.
• Discussion of challenges and open problems in the application of CT in an industrial setting.

Combinatorial and complementing methods

• Combinatorial analysis of existing test plans – analyzing test plans not generated by a CT algorithm in light of a combinatorial test space.
• Test plan reduction and completeness – both under stable, and under changing test requirements.
• CT and coverage metrics – combining the two, and studying the relation between them.

Disclaimer: All text on the IWCT website has been carefully checked. Nevertheless, the organizers cannot guarantee the correctness, completeness and topicality of the information. Liability of the organizers is therefore excluded. The links to other websites have been carefully chosen. However, since the organizers have no say on their content, they assumes no responsibility for these.