ICST 2027
Mon 17 - Fri 21 May 2027 San Sebastian, Spain
ICST 2027 (series) /

Challenge Competition TrackICST 2027

ICST 2027 - Challenge Competition Track

20th IEEE International Conference on Software Testing, Verification and Validation (ICST) 2027
San Sebastián, Spain

Overview

The ICST 2027 Challenge Competition Track is a new initiative aimed at strengthening the connection between software testing research and real-world challenges from both academia and industry. The track provides a venue for organizing and participating in competitions that foster innovation, reproducibility, and empirical evaluation of software testing techniques on realistic and impactful problems.

Unlike traditional research papers, competitions encourage the community to compare approaches on common benchmarks, datasets, and reproducible evaluation infrastructures. They provide an opportunity to advance the state of the art through open challenges, shared artifacts, transparent evaluation methodologies, and fair experimental comparison.

The Challenge Competition Track welcomes established competitions that have been successfully run in previous editions of ICST and SBFT (International Workshop on Search-Based and Fuzz Testing). A special place is given to SBFT competitions, considering that traditionally, SBFT has been the place for such initiatives for a long time (co-located with ICSE - International Conference on Software Engineering).

Examples of suitable competition topics include, but are not limited to:

  • AI-enabled Software Testing
  • Large Language Models for Software Testing
  • Autonomous Systems Testing (Self-driving Cars, UAVs, Robotics)
  • Regression Testing
  • Test Generation
  • Test Prioritization
  • Mutation Testing
  • Fuzz Testing
  • Security and Vulnerability Testing
  • Mobile and Web Application Testing
  • Continuous Integration and DevOps Testing
  • Cyber-Physical Systems Testing
  • Testing Benchmarks and Datasets
  • Industrial Software Testing Challenges

Competitions should provide participants with clearly defined tasks, benchmark systems or datasets, transparent evaluation criteria, and a reproducible execution infrastructure that enables fair comparison of submitted approaches.

Call for Competition Proposals

Researchers and practitioners are invited to submit proposals for organizing a competition as part of the ICST 2027 Challenge Competition Track.

Each proposal should describe:

  • motivation and relevance of the challenge;
  • competition objectives and research questions;
  • target participants and expected community;
  • competition task(s);
  • benchmark systems, datasets, or software artifacts;
  • evaluation methodology and ranking criteria;
  • competition infrastructure (e.g., GitHub repository, benchmark platform, Docker containers, cloud infrastructure);
  • reproducibility strategy;
  • expected timeline;
  • organizing committee and previous experience.

Preference will be given to competitions that:

  • address important and timely software testing challenges;
  • promote reproducibility and open science;
  • provide publicly available datasets and evaluation artifacts whenever possible;
  • encourage participation from both academia and industry;
  • demonstrate technical feasibility and long-term sustainability;
  • have the potential to become recurring community benchmarks;
  • and are complementary in topics to existing one.

Proposal Format

Competition proposals should be 4 pages, including references, and follow the ICST formatting guidelines.

Evaluation of Competition Proposals

Competition proposals will be evaluated by the Challenge Competition Track Committee according to the following criteria:

  • relevance to the software testing community;
  • novelty and expected scientific impact;
  • technical quality of the proposed challenge;
  • soundness of the evaluation methodology;
  • quality of the benchmark systems and datasets;
  • reproducibility of the proposed infrastructure;
  • fairness and transparency of the evaluation process;
  • expected community engagement;
  • feasibility of the organization;
  • long-term sustainability and reusability of the competition artifacts.

Accepted competition organizers will be responsible for organizing and running the competition, evaluating participant submissions, and presenting the results during ICST 2027.

Requirements for Accepted Competitions

Accepted competitions are expected to provide a complete competition package that enables fair, transparent, and reproducible evaluation.

Competition Website

Each competition should maintain a public website including, as examples:

The website should include:

  • competition overview and objectives;
  • important dates;
  • participation instructions;
  • submission guidelines;
  • evaluation methodology;
  • ranking procedure;
  • contact information.

Competition Repository

Each competition should provide a public repository, e.g., GitHub, containing, whenever applicable:

  • competition documentation;
  • benchmark systems;
  • datasets;
  • example submissions;
  • APIs and interfaces;
  • FAQs and issue tracker;
  • updates and announcements.

Reproducibility Infrastructure

Competition organizers are encouraged to provide reproducible evaluation artifacts, including where appropriate, as in the example: https://github.com/christianbirchler-org/sdc-testing-competition

  • Docker or OCI containers;
  • virtual machine images;
  • benchmark runners;
  • automated evaluation scripts;
  • execution workflows;
  • installation instructions;
  • documentation for reproducing the evaluation.

The objective is to ensure that competition results can be independently reproduced and that the competition can be maintained in future editions.

Evaluation Process

Competition organizers are responsible for:

  • validating submitted systems;
  • executing the evaluation;
  • ensuring fairness and confidentiality throughout the evaluation process;
  • publishing rankings;
  • documenting the evaluation methodology;
  • making evaluation results available to participants.

Participation in Accepted Competitions

Each accepted competition will publish its own detailed call for participants, including:

  • competition objectives;
  • submission instructions;
  • deadlines;
  • benchmark description;
  • technical requirements;
  • evaluation methodology;
  • ranking criteria.

Participants may choose to enter one or multiple competitions within the Challenge Competition Track.

Competition organizers may define additional submission requirements specific to their domain, provided these requirements are clearly documented before the competition opens.

Organizers are encouraged to maintain public repositories and discussion forums to facilitate community participation and ensure long-term availability of competition artifacts.

Competition Reports and Participant Papers

Each accepted competition may produce two types of publications.

Competition Report

Competition organizers will prepare a Competition Report describing:

  • motivation;
  • competition design;
  • benchmark systems and datasets;
  • evaluation methodology;
  • competition infrastructure;
  • participating teams;
  • experimental results;
  • lessons learned;
  • future directions.

Competition Reports should be up to 8 pages, including references. Typically, this is 4 pages, as in this example: https://doi.org/10.1109/ICST62969.2025.10989045. They will be included in the ICST proceedings, subject to the conference publication policies.

Participant Papers

Competition organizers may invite participating teams to submit short papers describing their approaches.

Participant papers should describe:

  • the proposed technique;
  • implementation details;
  • experimental evaluation;
  • lessons learned.

The recommended length is 2-4 pages, including references. See example of tool/approach report: https://doi.org/10.1109/ICST62969.2025.10988989. Organizers may adapt this according to the needs of their competition, subject to ICST publication guidelines.

Presentation at ICST 2027

Accepted competitions will be presented during dedicated sessions at ICST 2027.

Depending on the number and nature of accepted competitions, the program may include:

  • presentations by competition organizers and an opening talk;
  • presentations by top-ranked participants;
  • award announcements;
  • demonstrations;
  • panel discussions.

The Challenge Competition Track aims to promote collaboration between academia and industry while fostering reusable benchmarks and sustainable evaluation infrastructures for the software testing community.

Important Dates

Tentative - to be updated according to the official ICST 2027 schedule.

Activity Date
Competition Proposal Submission October 2, 2026
Proposal Notification October 20, 2026
Competition Website and Repository Available 31 October, 2026
Competition Opens 1st November, 2026
Participant Submission Deadline 15 January, 2027
Notification to Participants 15 of February, 2027
Tool papers and the Competition Reports Submission 28 February, 2027
Camera-ready or Tool papers and the Competition Reports March 7th, 2027

Organizing Committee

Challenge Competition Track Chairs

Challenge Competition Evaluation Committee

Contacts

For questions regarding the ICST 2027 Challenge Competition Track, please contact the Track Chairs.

We look forward to receiving innovative competition proposals that address challenging software testing problems, promote reproducible research, and strengthen collaboration between academia and industry.

Questions? Use the ICST Challenge Competition Track contact form.
Program Comittee