IWCT 2023 is to be held in conjunction with ICST 2023, focusing on combinatorial testing. The workshop welcomes academic research submissions, as well as industrial experience reports.

Combinatorial Testing (CT), or Combinatorial Interaction Testing (CIT), is a widely applicable generic methodology and technology for software verification and validation, considered a testing best practice. In a combinatorial test plan, all interactions between parameters up to a certain level are covered. For example, in pairwise testing, for every pair of parameters, every pair of values will appear at least once. Studies show that CT is more efficient and effective than random testing.

CT has gained significant interest in recent years, both in research and in practice. However, many issues still remain unresolved, and much research is still needed in the field. For example, while pairwise testing is a well recognized and popular test planning method, investigations of actual failures in a number of software and systems convincingly show that pairwise testing is usually not sufficient, so high strength CT (i.e., t-way for t > 2) may be needed.

In addition, the combinatorial test plans need to exclude invalid combinations of test values that cannot be executed, which limits the degrees of freedom the algorithms have, thus complicating the problem. Moreover, modeling languages and tools for easily capturing the input test space are also required for real-life applicability of CT. Other obstacles for wide acceptance of CT in industry are the gap between the generated test plans and executable tests, and the difficulty in determining expected results for the generated tests. Finally, empirical studies on CT, as well as thorough comparison with other methods, are also required.

In this workshop, we plan to bring together researchers actively working on combinatorial testing, and create a productive and creative environment for sharing and collaboration. IWCT being the primary venue dedicated to CT, yet there are many researchers working in the field, we expect, like in previous years, to see high responsiveness to take part in the workshop. Researchers attending the workshop will have an opportunity to publish their work in a dedicated venue, create new collaborations and take active part in the growing community of researchers working in the field.

The workshop will also be a meeting place between academia and industry, thus uniting academic excellence and industrial experience and needs. This will allow participants from academia to learn about the industrial experience in practical application of CT to real-life testing problems, and together with the colleagues from industry identify the difficulties that are obstacle to wider application of CT, and should be addressed in future research. Industrial participants will have an opportunity to meet the leading scientists in the field, and hear about the latest advances and innovations.

Combinatorial test competition

IWCT will host the second edition of the competition among tools for combinatorial testing - We are still working on the details, some preliminary information is here: https://fmselab.github.io/ct-competition/. We are working on extending the benchmarks with new categories, please stay tuned.

Previous Editions of IWCT

We are proud of more than ten previous editions of IWCT, showing the maturity of the field and the support of our community:

11th IWCT 2022

10th IWCT 2021

9th IWCT 2020

8th IWCT 2019

7th IWCT 2018

6th IWCT 2017

5th IWCT 2016

4th IWCT 2015

3rd IWCT 2014

2nd IWCT 2013

1st IWCT 2012

Disclaimer: All text on the IWCT website has been carefully checked. Nevertheless, the organizers cannot guarantee the correctness, completeness and topicality of the information. Liability of the organizers is therefore excluded. The links to other websites have been carefully chosen. However, since the organizers have no say on their content, they assumes no responsibility for these.

Accepted Papers

Call for Papers

We invite submissions of high-quality papers presenting original work on both theoretical and experimental aspects of combinatorial testing (CT).

Each submitted paper must conform to the IEEE two-column publication format. All papers will be reviewed by at least three members from the program committee. If a submission is accepted, at least one author of the paper is required to attend the workshop and present the paper for it to be published in the ICST 2023 workshop proceedings.

Submission

Submissions are handled via an EasyChair instance provided by ICST for all workshops, authors are kindly asked to select the track ‘International Workshop on Combinatorial Testing’ in the first step: https://easychair.org/conferences/?conf=icstw23

Full and short papers

We invite submissions for full papers (up to 8 pages, excluding references) as well as short papers (up to 4 pages, excluding references). For both of these submission types, pages are not a strict limit, if you need more space please contact the chairs. Accepted papers in these two categories – full and short – will be part of the proceedings of IWCT 2023.

Journal-First

The aim of the Journal-First (JF) submission category is to further enrich the program of IWCT, as well as to provide an overall more flexible path to publication and dissemination of original research that is within the scope of IWCT. A submission in this category must adhere to the following criteria:

1. It should be clearly within the scope of the workshop.
2. It should be recent: it should have been accepted and made publicly available in a journal (online or in print) by January 1, 2021 or more recently.
3. It has not been presented at, and is not under consideration for, Journal- First tracks of other conferences or workshops.
4. The submission has to be in the form of a 2-page extended abstract and has to provide a concise summary of the published journal paper.

Journal-First submissions must be marked as such in the submission’s title, and must explicitly include full bibliographic details (including a DOI) of the journal publication they are based on. Since the referenced journal papers have already been reviewed and accepted by the corresponding journals, submissions in the JF category will not be reviewed again for technical content. Submissions will be judged on the basis of the above criteria, but also considering how well they would complement the workshop’s technical program. Accepted submissions in this category will be part of the proceedings of IWCT 2023, with the title equal to the original title of the article with the prefix Summary of.

Posters

In addition to the traditional program topics from the past years, and taking advantage of the fact that some of the lead CT tool developers have expressed interest in participating again in the second iteration of the CT competition (see below), we would like to offer with the poster session a low-threshold opportunity – especially targeting industry – to present activities within the scope of the workshop to the participants of IWCT. A tool demonstration can be given along a poster. In case a poster contribution wishes to provide more details or information, but does not want to submit a short of full paper, the authors can choose to submit in addition to the poster a maximal 2-page extended abstract detailing the content of the poster. The title of the extended abstract has to have the prefix Extended Abstract of Poster and these extended abstracts – if accepted – will also appear in the proceedings of IWCT 2023.

CT competition

IWCT 2023 will also host the 2nd combinatorial testing competition for combinatorial test generation tools. Details for how to participate can be found here.

Special Issue in Section of Springer Nature Journal

We plan to organize a call among the best papers of the workshop to submit extended versions in the Springer Nature Computer Science journal in the Section Combinatorial Methods and Models in System Testing. More information will be provided at the workshop.

Topics for submissions

Topics of interest for submissions include, but are not limited to:

Combinatorial testing workflow

• Modeling the input space for CT – the input to CT algorithms is a set of parameters, respective values, and constraints on value combinations. This input should capture correctly the points of variability in testing the system. While this input is clearly crucial for the effectiveness of CT, it is a difficult problem.
• Efficient algorithms to generate test suites with small size for t-way testing for t > 2, involving support of constraints on combinations that are possible.
• Determination of expected system behavior for each test case – while the test cases are automatically generated by the CT algorithm, determining the expected system behavior is currently usually a manual task.
• Executing CT test suites – the result of CT algorithms is a list of tests, where a test is represented by a value for each parameter. There is still significant effort in transforming this representation into actual tests that a tester or testing tool can execute.
• Combinatorial testing based fault localization.
• Implementation of CT in existing testing infrastructures.
• Handling changes in test requirements – current CT methods focus on onetime generation of a test plan from a given set of requirements. However, test requirements are almost never static, and tend to change between different releases and versions. On the other hand, generating a new set of tests for each release may not be practical either.

Real-world experience in deployment of combinatorial testing

• Empirical studies and feedback from practical applications of CT.
• Evaluation and return of investment metrics to assess the degree of usefulness of CT.
• Methodology used for test space modeling and determination of interaction coverage requirements.
• Discussion of challenges and open problems in the application of CT in industrial settings.

Applicability of combinatorial testing

• Comparison and combination of CT with other dynamic verification methods.
• Investigation of historical records of failures to determine the kind of CT which may have detected underlying faults.
• Combinatorial testing for concurrent and real-time systems.
• CT for testing cloud computing systems and use of combinatorial methods in cloud architecture.
• Combinatorial testing for AI-based systems.
• Application of CT in other domains, e.g., information security, study of gene regulation and other biotechnology applications and mechanical engineering.
• Combinatorial testing of feature models for software product lines.

Combinatorial and complementing methods

• Combinatorial analysis of existing test suites – analyze a test suite not generated by a CT algorithm in light of a combinatorial test space.
• Test plan reduction and completeness – both under stable, and under changing test requirements.
• CT and coverage metrics – combining the two, and studying the relation between them.

Disclaimer: All text on the IWCT website has been carefully checked. Nevertheless, the organizers cannot guarantee the correctness, completeness and topicality of the information. Liability of the organizers is therefore excluded. The links to other websites have been carefully chosen. However, since the organizers have no say on their content, they assumes no responsibility for these.

Accepted (Long/Short) papers

• Gang Qin, Junjun Zheng and Tatsuhiro Tsuchiya. Meta-Heuristic Algorithm for Constructing Higher-Index Covering Arrays for Combinatorial Interaction Testing
• Andrea Bombarda and Angelo Gargantini. Incremental generation of combinatorial test suites starting from existing seed tests
• Tyler Cody and Laura Freeman. Metric Learning Improves the Ability of Combinatorial Coverage Metrics to Anticipate Classification Error
• Simon Diemert, Adam Casey and Jeremiah Robertson. Challenging Autonomy with Combinatorial Testing
• Hanefi Mercan and Cemal Yilmaz. Hints in Unified Combinatorial Interaction Testing
• Krishna Khadka, Jaganmohan Chandrasekaran, Yu Lei, Raghu N. Kacker and Richard Kuhn. Synthetic Data Generation Using Combinatorial Testing and Variational Autoencoder
• Ludwig Kampel, Michael Wagner, Dimitris E. Simos, Mihai Nica, Dino Dodig, David Kaufmann and Franz Wotawa. Applying CT-FLA for AEB Function Testing: A Virtual Driving Case Study
• Rick Kuhn, M S Raunak and Raghu Kacker. Ordered t-way Combinations for Testing State-based Systems
• Jovan Zivanovic, Manuel Leithner, Dimitris E. Simos, Michael Pitzer and Peter J. Slanina. Combinatorial Methods for HTML Sanitizer Security Testing

Accepted Journal-First papers

• Ludwig Kampel, Paris Kitsos and Dimitris E. Simos. Summary of Locating Hardware Trojans using Combinatorial Testing for Cryptographic Circuits
• Bernhard Garn, Jovan Zivanovic, Manuel Leithner and Dimitris E. Simos. Summary of Combinatorial Methods for Dynamic Gray-Box SQL Injection Testing
• Bernhard Garn, Dominik-Philip Schreiber, Dimitris E. Simos, Rick Kuhn, Jeff Voas and Raghu Kacker. Summary of Combinatorial methods for testing Internet of Things smart home systems
• Michael Wagner, Charles Colbourn and Dimitris E. Simos. Summary of In-Parameter-Order strategies for covering perfect hash families

Dear Authors of IWCT and A-MOST,

We call for extended versions of submitted papers to the 19th Workshop on Advances in Model Based Testing (A-MOST 2023) and the 12th International Workshop on Combinatorial Testing (IWCT 2023), which were both held co-located with the 16th IEEE International Conference on Software Testing, Verification and Validation (ICST 2023) in Dublin, Ireland in April 2023, to be considered for publication as post-proceedings in a special issue of the journal SN Computer Science in the Section Combinatorial Methods and Models in System Testing. We will consider submissions containing at least 30% new material and authors should make sure to clearly explain the additional contribution in their submissions.

This SI of the journal ‘Springer Nature Computer Science’ in the ‘SECTION VI- Combinatorial Methods and Models in System Testing’ dedicated to post-proceedings of both IWCT and A-MOST 2023 with the title “Advances in Combinatorial and Model-based Testing 2023” is now officially listed on the journal’s homepage at (NEW submission deadline November 20, 2023):

https://www.springer.com/journal/42979/updates/17257234

The CFP is now also officially available on the journal’s homepage at: https://resource-cms.springernature.com/springer-cms/rest/v1/content/25867610/data/v1

In case of any questions, please don’t hesitate to reach out to us, we are happy to help!

Yours sincerely,

Cristina, Florian, Angelo and Bernhard

Your Chairs of A-MOST and IWCT