ICSME 2025
Sun 7 - Fri 12 September 2025 Auckland, New Zealand
Thu 11 Sep 2025 14:30 - 14:45 at Case Room 3 260-055 - Session 9 - Testing 3 Chair(s): Sigrid Eldh

Due to the large number of possible interactions and transitions among features in dynamically adaptive systems, testing such systems poses significant challenges. To verify that such systems behave correctly, combinatorial interaction testing (CIT) can create concise test suites covering all valid pairs of features of such systems. While CIT claims to find all errors caused by two features, it does not cover certain errors occurring only for specific transitions between features. To address this issue we study the technique of Combinatorial Transition Testing (CTT), which includes both generation and detection of what we call behavioural transition errors. From an initial generation algorithm that combines both interaction and transition coverage but lacks scalability, we propose an optimised version that enables CTT even for hundreds of features. From a valid test suite covering all transitions, we complete our testing approach with a test oracle that detects all behavioural transition errors without any prior knowledge of the system’s behaviour. After a comprehensive analysis over a large number of feature models, we conclude that size of CTT-generated test suites and test effort needed to use our test oracle are linearly correlated to CIT-generated ones and that CTT grows logarithmically in the number of features.

Thu 11 Sep

Displayed time zone: Auckland, Wellington change

13:30 - 15:00
Session 9 - Testing 3Journal First Track / NIER Track / Tool Demonstration Track / Research Papers Track / Registered Reports at Case Room 3 260-055
Chair(s): Sigrid Eldh Ericsson AB, Mälardalen University, Carleton University
13:30
15m
Full-paper
Metamorphic Testing of Large Language Models for Natural Language Processing
Research Papers Track
Steven Cho The University of Auckland, New Zealand, Stefano Ruberto JRC European Commission, Valerio Terragni University of Auckland
Pre-print
13:45
15m
Onweer: Automated Resilience Testing through Fuzzing
Research Papers Track
Gilles Coremans Vrije Universiteit Brussel, Coen De Roover Vrije Universiteit Brussel
Pre-print
14:00
10m
Generating Highly Structured Test Inputs Leveraging Constraint-Guided Graph Refinement
Registered Reports
Zhaorui Yang University of California, Riverside, Yuxin Qiu University of California at Riverside, Haichao Zhu Meta, Qian Zhang University of California at Riverside
14:10
10m
Prioritizing Test Smells: An Empirical Evaluation of Quality Metrics and Developer Perceptions
NIER Track
Md Arif Hasan University of Dhaka, Bangladesh, Toukir Ahammed Institute of Information Technology, University of Dhaka
14:20
10m
LLMShot: Reducing snapshot testing maintanence via LLMs
NIER Track
Ergün Batuhan Kaynak Bilkent University, Mayasah Lami Bilkent University, Sahand Moslemi Yengejeh Bilkent University, Anil Koyuncu Bilkent University
Pre-print
14:30
15m
Combinatorial Transition Testing in Dynamically Adaptive Systems: Implementation and Test Oracle
Journal First Track
Pierre Martou UCLouvain / ICTEAM, Benoît Duhoux Université catholique de Louvain, Belgium, Kim Mens Université catholique de Louvain, ICTEAM institute, Belgium, Axel Legay Université Catholique de Louvain, Belgium
14:45
10m
LLMLOOP: Improving LLM-Generated Code and Tests through Automated Iterative Feedback Loops
Tool Demonstration Track
Ravin Ravi University of Auckland, Dylan Bradshaw University of Auckland, Stefano Ruberto JRC European Commission, Gunel Jahangirova King's College London, Valerio Terragni University of Auckland
Pre-print