Accepted Papers
Title | |
---|---|
AMBER: AI-enabled Java Microbenchmark Harness Testing Tools and Data Showcase Pre-print | |
Codehacks: A Dataset of Adversarial Tests For Competitive Programming Problems Obtained From Codeforces Testing Tools and Data Showcase | |
E2E-Loader: A Tool to Generate Performance Tests from End-to-End GUI-level Tests Testing Tools and Data Showcase | |
RESTgym: A Flexible Infrastructure for Empirical Assessment of Automated REST API Testing Tools Testing Tools and Data Showcase | |
Rocket: A System-Level Fuzz-Testing Framework for the XRPL Consensus Algorithm Testing Tools and Data Showcase | |
ViMoTest: A Tool to Specify ViewModel-based GUI Test Scenarios using Projectional Editing Testing Tools and Data Showcase |
Call for Papers
The Testing Tools and Data Showcase Track seeks to bridge the gap between research and practice by focusing on software testing tools and their demonstrations that advance both the state of the art and the state of the practice. The track invites submissions that showcase tools at several stages of maturity, including promising research prototypes, widely used research tools, and commercial tools (only if they contribute to scientific knowledge), and from both academia and industry. In addition, the track aims to actively promote and recognize the creation of reusable datasets that are designed and built not only for a specific research project but for the testing community as a whole. These datasets should enable other practitioners and researchers to jumpstart their research efforts, and also allow the reproducibility of earlier work.
The ICST 2025 Testing Tools and Data Showcase Track will accept two types of submissions: 1) tool papers and (2) data showcase papers.
1 - Tool Papers
Tool papers should:
- Fall under the topics mentioned in the ICST 2025 research track.
- Present and discuss a tool that has NOT been published before as a tool paper.
- Motivate the need for the tool and clearly describe the complexity of the addressed problem.
- Discuss the tool’s goals, envisioned users and implied use case scenarios, requirements, implemented testing process or testing technique, solved technical challenges, and maturity level.
- Explain the tool’s overall architecture and its inner workings.
- Describe the tool’s novelty and how it relates to previous industrial or research efforts.
- Report on how the tool has been validated (e.g., from previously published research work or new experiments); experience reports from using the tool in industrial settings will be highly valued. Although tool papers are NOT expected to necessarily contain large-scale empirical studies, reporting any empirical result or user feedback is highly encouraged. In the case of early prototypes, illustrating the design of the planned validation studies is acceptable.
- Discuss any limitation of the tool as well as potential ways that the tool could be extended (by the authors or the community) in the future.
- Include a statement on the tool availability (see also below).
- Include (at the end of the abstract) the URL of a 3-to-5-minute screencast, either with annotations or voice-over, that provides a concise version of the tool demo scenario. The video should be posted on YouTube (as unlisted video) or hosted on the tool’s website. The main purpose of such a screencast is to show the functionality of the tool; it will not be reviewed as a form of long-term artifact
The tool itself should be made available (with an appropriate license notice) at the time of submission of the paper for review. At a minimum, the tool should be accessible (either free to download or online accessible). If possible, the source code of the tool should also be available. Exceptions can be granted only if a valid reason is provided explaining why the tool cannot be released (e.g., organizational rules, Intellectual Property restrictions). The tool should include clear installation instructions and an example dataset that allow the reviewers to run the tool
Upon acceptance, authors of papers that mentioned “publicly available” (or equivalent) under “Tool availability” should archive the tool on a persistent repository that can provide a digital object identifier (DOI) such as zenodo.org, figshare.com, IEEE DataPort, or institutional repositories. In addition, the DOI-based citation of the tool should be included in the camera-ready version of the paper.
2 - Data Showcase Papers
Data Showcase papers are descriptions of datasets relevant to the ICST 2025 topics, which can be used by other practitioners or researchers.
Data showcase papers should include:
- A description of the data source.
- A description of the methodology used to gather the data (including provenance and the tool used to create/generate/gather the data, if any).
- A description of the storage mechanism, including a schema if applicable.
- If the data has been used by the authors or others, a description of how this was done, including references to previously published papers.
- A description of the originality of the dataset (that is, even if the dataset has been used in a published paper, its complete description must be unpublished) and similar existing datasets (if any).
- Ideas for future research questions that could be answered using the dataset.
- Ideas for further improvements that could be made to the dataset.
- Any limitations and/or challenges in creating or using the dataset.
The dataset should be made available (with an appropriate license notice) at the time of submission of the paper for review. The dataset should include detailed instructions about how to use the dataset (e.g., how to import the data or how to access the data once it has been imported).
At a minimum, upon acceptance of the paper, the authors should archive the data on a persistent repository that can provide a digital object identifier (DOI) such as zenodo.org, figshare.com, IEEE DataPort, or institutional repositories. In addition, the DOI-based citation of the dataset should be included in the camera-ready version of the paper.
If custom tools have been used to create the dataset, we expect the paper to be accompanied by the source code of the tools, along with clear documentation on how to run the tools to recreate the dataset. The tools should be open source, accompanied by an appropriate license; the source code should be citable, i.e., refer to a specific release and have a DOI. If you cannot provide the source code or the source code clause is not applicable (e.g., because the dataset consists of qualitative data), please provide a short explanation of why this is not possible.
Evaluation:
Tool papers will be evaluated based on:
- The relevance and significance of the addressed problem.
- The innovation element of the approach.
- The availability, maturity, and adoption of the tool.
- The presence of lessons learned from developing or using the tool.
- The quality of the presentation.
Data Showcase papers will be evaluated based on:
- value, usefulness, and reusability of the datasets.
- quality of the presentation.
- availability of the datasets.
Submission
Submissions will be handled via EasyChair (ICST2025 / Testing Tools and Data Showcase Track) at https://easychair.org/conferences/?conf=icst2025.
The Testing Tools and Data Showcase track of ICST 2025 uses single-anonymous reviewing, meaning authors and tools do not have to be anonymized. All submissions must:
- Be in PDF and conform to the the IEEE Conference Proceedings Formatting Guidelines. Templates for LaTeX and Word are available at: http://www.ieee.org/conferences_events/conferences/publishing/templates.html.
- Not exceed 5 pages, including all text, figures, tables, appendices, and references.
- Not have been published elsewhere or be under review elsewhere while under review for ICST 2025.
- Comply with IEEE plagiarism policy as well as IEEE Policy on Authorship.
To make research datasets and accessible and citable, we further encourage authors to attend to the FAIR rules, i.e., data should be: Findable, Accessible, Interoperable, and Reusable.
Submissions that are not in compliance with the required submission format or that are out of the scope of the track will be rejected without being reviewed.
All authors, reviewers, and organizers are expected to uphold the IEEE Code of Conduct.
Publication and Presentation
Accepted papers will be published as part of the conference proceedings. Camera-ready and presentation details will be provided after notification of acceptance. At least one author of each accepted paper must register for the conference and present the paper at the conference.
In addition to delivering a presentation that will be included in the conference program, authors of accepted tool papers will have the opportunity to conduct a hands-on session where attendees of ICST 2025 can actively use and experiment with the demonstrated tools.
Important Dates
- Full paper: December 4, 2024
- Author notification: January 19, 2025
- Camera Ready: February 13, 2025
This program is tentative and subject to change.
Wed 2 AprDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
11:00 - 12:30 | LLMs in TestingResearch Papers / Industry / Journal-First Papers at Aula Magna (AM) Chair(s): Phil McMinn University of Sheffield | ||
11:00 15mTalk | AugmenTest: Enhancing Tests with LLM-driven Oracles Research Papers Shaker Mahmud Khandaker Fondazione Bruno Kessler, Fitsum Kifetew Fondazione Bruno Kessler, Davide Prandi Fondazione Bruno Kessler, Angelo Susi Fondazione Bruno Kessler Pre-print | ||
11:15 15mTalk | Impact of Large Language Models of Code on Fault Localization Research Papers Suhwan Ji Yonsei University, Sanghwa Lee Kangwon National University, Changsup Lee Kangwon National University, Yo-Sub Han Yonsei University, Hyeonseung Im Kangwon National University, South Korea | ||
11:30 15mTalk | An Analysis of LLM Fine-Tuning and Few-Shot Learning for Flaky Test Detection and Classification Research Papers | ||
11:45 15mTalk | Evaluating the Effectiveness of LLMs in Detecting Security Vulnerabilities Research Papers Avishree Khare , Saikat Dutta Cornell University, Ziyang Li University of Pennsylvania, Alaia Solko-Breslin University of Pennsylvania, Mayur Naik UPenn, Rajeev Alur University of Pennsylvania | ||
12:00 15mTalk | FlakyFix: Using Large Language Models for Predicting Flaky Test Fix Categories and Test Code Repair Journal-First Papers Sakina Fatima University of Ottawa, Hadi Hemmati York University, Lionel Briand University of Ottawa, Canada; Lero centre, University of Limerick, Ireland | ||
12:15 15mTalk | Integrating LLM-based Text Generation with Dynamic Context Retrieval for GUI Testing Industry Juyeon Yoon Korea Advanced Institute of Science and Technology, Seah Kim Samsung Research, Somin Kim Korea Advanced Institute of Science and Technology, Sukchul Jung Samsung Research, Shin Yoo Korea Advanced Institute of Science and Technology |
14:00 - 15:30 | Test OracleJournal-First Papers / Research Papers / Industry at Aula Magna (AM) Chair(s): Mike Papadakis University of Luxembourg | ||
14:00 15mTalk | Differential Testing of Concurrent Classes Research Papers Valerio Terragni University of Auckland, Shing-Chi Cheung Hong Kong University of Science and Technology | ||
14:15 15mTalk | Hybrid Equivalence/Non-Equivalence Testing Research Papers Laboni Sarker University of California at Santa Barbara, Tevfik Bultan University of California at Santa Barbara | ||
14:30 15mTalk | Metamorphic Testing for Pose Estimation Systems Research Papers Matias Duran Trinity College Dublin, Ireland, Thomas Laurent Lero@Trinity College Dublin, Ellen Rushe School of Computing Dublin City University, Anthony Ventresque Trinity College Dublin & Lero, Ireland | ||
14:45 15mTalk | Mutation-Guided Metamorphic Testing of Optimality in AI Planning Journal-First Papers Quentin Mazouni Simula Research Laboratory, Arnaud Gotlieb Simula Research Laboratory, Helge Spieker Simula Research Laboratory, Mathieu Acher University of Rennes - Inria - CNRS - IRISA, Benoit Combemale University of Rennes, Inria, CNRS, IRISA Link to publication DOI | ||
15:00 15mTalk | LLMs in the Heart of Differential Testing: A Case Study on a Medical Rule Engine Industry Erblin Isaku Simula Research Laboratory, and University of Oslo (UiO), Christoph Laaber Simula Research Laboratory, Hassan Sartaj Simula Research Laboratory, Shaukat Ali Simula Research Laboratory and Oslo Metropolitan University, Thomas Schwitalla Cancer Registry of Norway, Jan F. Nygård Cancer Registry of Norway |
14:00 - 15:30 | |||
14:00 15mTalk | Accessible Smart Contracts Verification: Synthesizing Formal Models with Tamed LLMs Industry Jan Corazza TU Dortmund University, Ivan Gavran Informal Systems, Gabriela Moreira Informal Systems, Danel Neider TU Dortmund University | ||
14:15 15mTalk | Evaluation of the Choice of LLM in a Multi-agent solution for GUI-test Generation Industry Stevan Tomic Blekinge Institute of Technology, Emil Alégroth Blekinge Institute of Technology, Maycel Isaac Synteda | ||
14:30 15mTalk | Assessing the Uncertainty and Robustness of the Laptop Refurbishing Software Industry Chengjie Lu Simula Research Laboratory and University of Oslo, Jiahui Wu Simula Research Laboratory and University of Oslo, Shaukat Ali Simula Research Laboratory and Oslo Metropolitan University, Mikkel Labori Olsen Danish Technological Institute | ||
14:45 15mTalk | FuzzE, Development of a Fuzzing Approach for Odoo's Tours Integration Testing Plateform Industry Gabriel Benoit University of Namur, François Georis Odoo S.A., Géry Debongnie Odoo S.A., Benoît Vanderose University of Namur, Xavier Devroey University of Namur Pre-print | ||
15:00 15mTalk | Early V&V in Knowledge-Centric Systems Engineering: Advances and Benefits in Practice Industry Jose Luis de la Vara Universidad de Castilla - La Mancha, Juan Manuel Morote Independent Researcher, Clara Ayora Universidad de Castilla-La Mancha, Giovanni Giachetti Universidad Andrés Bello Chile, Luis Alonso The REUSE Company, Roy Mendieta The REUSE Company, David Muñoz The REUSE Company, Ricardo Ruiz Nolasco RGB Medical Devices, Antonio González RGB Medical Devices S.A. | ||
15:15 15mTalk | Practical Pipeline-Aware Regression Test Optimization for Continuous Integration Industry Daniel Schwendner BMW Group, Maximilian Jungwirth BMW Group, University of Passau, Martin Gruber BMW Group, University of Passau, Martin Knoche BMW Group, Daniel Merget BMW Group, Gordon Fraser University of Passau |
16:00 - 17:00 | Test Case Selection, Prioritisation, ReductionResearch Papers / Industry at Aula Magna (AM) Chair(s): Andrea Stocco Technical University of Munich, fortiss | ||
16:00 15mTalk | The Impact of List Reduction for Language Agnostic Test Case Reducers Research Papers Tobias Heineken Friedrich-Alexander-Universität Erlangen-Nürnberg, Michael Philippsen Friedrich-Alexander Universität Erlangen-Nürnberg (FAU), Programming Systems Group | ||
16:15 15mTalk | RustyRTS: Regression Test Selection for Rust Research Papers Simon Hundsdorfer Technical University of Munich, Roland Würsching Technical University of Munich, Alexander Pretschner TU Munich | ||
16:30 15mTalk | ML-Based Test Case Prioritization: A Research and Production Perspective in CI Environments Industry Md Asif Khan Ontario Tech University, Akramul Azim Ontario Tech University, Ramiro Liscano Ontario Tech University, Kevin Smith International Business Machines Corporation (IBM), Yee-Kang Chang International Business Machines Corporation (IBM), Gkerta Seferi International Business Machines Corporation (IBM), Qasim Tauseef International Business Machines Corporation (IBM) | ||
16:45 15mTalk | Evaluating Machine Learning-Based Test Case Prioritization in the Real World: An Experiment with SAP HANA Industry Jeongki Son SAP Labs Korea, Gabin An Roku, Jingun Hong SAP Labs, Shin Yoo Korea Advanced Institute of Science and Technology |
Thu 3 AprDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
10:30 - 11:00 | |||
10:30 30mCoffee break | Break Social |
11:00 - 12:30 | Testing ML Systems and Fault LocalisationIndustry / Research Papers at Aula Magna (AM) Chair(s): Atif Memon Apple | ||
11:00 15mTalk | On Accelerating Deep Neural Network Mutation Analysis by Neuron and Mutant Clustering Research Papers Pre-print | ||
11:15 15mTalk | Benchmarking Image Perturbations for Testing Automated Driving Assistance Systems Research Papers Stefano Carlo Lambertenghi Technische Universität München, fortiss GmbH, Hannes Leonhard Technical University of Munich, Andrea Stocco Technical University of Munich, fortiss Pre-print | ||
11:30 15mTalk | Turbulence: Systematically and Automatically Testing Instruction-Tuned Large Language Models for Code Research Papers Shahin Honarvar Imperial College London, Mark van der Wilk University of Oxford, Alastair F. Donaldson Imperial College London | ||
11:45 15mTalk | Taming Uncertainty for Critical Scenario Generation in Automated Driving Industry Selma Grosse DENSO Automotive GmbH, Dejan Nickovic Austrian Institute of Technology, Cristinel Mateis AIT Austrian Institute of Technology GmbH, Alessio Gambi Austrian Institute of Technology (AIT), Adam Molin DENSO AUTOMOTIVE | ||
12:00 15mTalk | Multi-Project Just-in-Time Software Defect Prediction Based on Multi-Task Learning for Mobile Applications Research Papers Feng Chen Chongqing University of Posts and Telecommunications, Ke Yuxin Chongqing University of Posts and Telecommunications, Liu Xin Chongqing University of Posts and Telecommunications, Wei Qingjie Chongqing University of Posts and Telecommunications | ||
12:15 15mTalk | Fault Localization via Fine-tuning Large Language Models with Mutation Generated Stack Traces Industry Neetha Jambigi University of Cologne, Bartosz Bogacz SAP SE, Moritz Mueller SAP SE, Thomas Bach SAP, Michael Felderer German Aerospace Center (DLR) & University of Cologne |
11:00 - 12:30 | Testing Tools and Data ShowcaseTesting Tools and Data Showcase at Room A1 Chair(s): Domenico Bianculli University of Luxembourg | ||
11:00 15mTalk | Rocket: A System-Level Fuzz-Testing Framework for the XRPL Consensus Algorithm Testing Tools and Data Showcase Wishaal Kanhai , Ivar van Loon , Yuraj Mangalgi , Thijs van der Valk , Lucas Witte Delft University of Technology, Annibale Panichella Delft University of Technology, Mitchell Olsthoorn Delft University of Technology, Burcu Kulahcioglu Ozkan Delft University of Technology | ||
11:15 15mTalk | RESTgym: A Flexible Infrastructure for Empirical Assessment of Automated REST API Testing Tools Testing Tools and Data Showcase Davide Corradini University of Luxembourg, Michele Pasqua University of Verona, Mariano Ceccato University of Verona | ||
11:30 15mTalk | ViMoTest: A Tool to Specify ViewModel-based GUI Test Scenarios using Projectional Editing Testing Tools and Data Showcase Mario Fuksa University of Stuttgart, Sandro Speth Institute of Software Engineering, University of Stuttgart, Steffen Becker University of Stuttgart | ||
11:45 15mTalk | E2E-Loader: A Tool to Generate Performance Tests from End-to-End GUI-level Tests Testing Tools and Data Showcase Sergio Di Meglio Università degli Studi di Napoli Federico II, Luigi Libero Lucio Starace Università degli Studi di Napoli Federico II, Sergio Di Martino Università degli Studi di Napoli Federico II | ||
12:00 15mTalk | AMBER: AI-enabled Java Microbenchmark Harness Testing Tools and Data Showcase Antonio Trovato University of Salerno, Luca Traini University of L'Aquila, Federico Di Menna University of L'Aquila, Dario Di Nucci University of Salerno Pre-print | ||
12:15 15mTalk | Codehacks: A Dataset of Adversarial Tests For Competitive Programming Problems Obtained From Codeforces Testing Tools and Data Showcase Max Hort Simula Research Laboratory, Leon Moonen Simula Research Laboratory and BI Norwegian Business School |
Fri 4 AprDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
11:00 - 12:30 | Automated TestingIndustry / Research Papers / Journal-First Papers / Education at Aula Magna (AM) Chair(s): Cristian Cadar Imperial College London | ||
11:00 15mTalk | Testing Practices, Challenges, and Developer Perspectives in Open-Source IoT Platforms Research Papers Daniel Rodriguez-Cardenas , Safwat Ali Khan George Mason University, Prianka Mandal William & Mary, Adwait Nadkarni William & Mary, Kevin Moran University of Central Florida, Denys Poshyvanyk William & Mary Pre-print | ||
11:15 15mTalk | Many-Objective Neuroevolution for Testing Games Research Papers Patric Feldmeier University of Passau, Katrin Schmelz University of Passau, Gordon Fraser University of Passau Pre-print | ||
11:30 15mTalk | Black-Box Testing for Practitioners Education Matthias Hamburg IEEE Computer Society; istqb.org, Adam Roman Jagiellonian University, Faculty of Mathematics and Computer Science; International Software Testing Qualifications Board | ||
11:45 15mTalk | CUBETESTERAI: Automated JUnit Test Generation using the LLaMA Model Industry Daniele Gorla Department of Computer Science, Sapienza University of Rome, Shivam Kumar , Pietro Nicolaus Roselli Lorenzini , Alireza Alipourfaz | ||
12:00 15mTalk | Can Search-Based Testing with Pareto Optimization Effectively Cover Failure-Revealing Test Inputs? Journal-First Papers Lev Sorokin Technische Universität München, Germany, Damir Safin fortiss, Shiva Nejati University of Ottawa | ||
12:15 15mTalk | [prerecorded] ADGE: Automated Directed GUI Explorer for Android Applications Research Papers Yue Jiang Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China, Xiaobo Xiang Singular Security Lab, Beijing, China, Qingli Guo Institute of Information Engineering, Chinese Academy of Sciences, Qi Gong Key Laboratory of Network Assessment Technology, Institute of Information Engineering, Chinese Academy of Sciences, China, Xiaorui Gong Institute of Information Engineering, Chinese Academy of Science |
14:00 - 15:28 | Advances in Software TestingResearch Papers / Short Papers, Vision and Emerging Results / Industry / Education at Room A1 Chair(s): Annibale Panichella Delft University of Technology | ||
14:00 15mTalk | Code, Test and Coverage Evolution in Mature Software Systems: Changes over the Past Decade Research Papers | ||
14:15 15mTalk | Coverage Metrics for T-Wise Feature Interactions Research Papers Sabrina Böhm University of Ulm, Germany, Tim Jannik Schmidt University of Ulm, Paderborn University, Germany, Sebastian Krieter TU Braunschweig, Germany, Tobias Pett Karlsruhe Institute for Technology (KIT), Germany, Thomas Thüm TU Braunschweig, Germany, Malte Lochau University of Siegen | ||
14:30 15mTalk | LLM-based Labelling of Recorded Automated GUI-based Test cases Industry Diogo Buarque Franzosi Blekinge Institute of Technology, Emil Alégroth Blekinge Institute of Technology, Maycel Isaac Synteda | ||
14:45 15mTalk | A Tool-assisted Training Approach for Empowering Localization and Internationalization Testing Proficiency Education Maria Couto Federal University of Pernambuco, Breno Miranda Centro de Informática at Universidade Federal de Pernambuco, Kiev Gama Universidade Federal de Pernambuco | ||
15:00 7mTalk | Pre-trained Models for Bytecode Instructions Short Papers, Vision and Emerging Results Donggyu Kim Handong Global University, Taemin Kim , Jiho Shin York University, Song Wang York University, Heeyoul Choi , Jaechang Nam Handong Global University | ||
15:07 7mTalk | EnCus: Customizing Search Space for Automated Program Repair Short Papers, Vision and Emerging Results Seongbin Kim , Sechang Jang , Jindae Kim Seoul National University of Science and Technology, Jaechang Nam Handong Global University | ||
15:14 7mTalk | Test Generation from Use Case Specifications for IoT Systems: Custom, LLM-Based, and Hybrid Approaches Short Papers, Vision and Emerging Results Zacharie Chenail-Larcher École de technologie supérieure (ÉTS), Jean Baptiste Minani Concordia University, Montreal, Canada, Naouel Moha École de Technologie Supérieure (ETS) |