Competition Overview

Self-driving cars (SDCs) are equipped with onboard cameras and various sensors, which have already demonstrated the possibility of autonomous driving in real environments, leading to great interest in various application domains. However, despite the necessity of systematically testing such complex and automated systems to ensure their safe operation in real-world environments, there has been relatively limited investment in this direction so far.

The SDC Testing Competition organized jointly by the International Conference on Software Testing, Verification and Validation (ICST) and Search-Based and Fuzz Testing (SBFT) workshop @ ICSE is an initiative designed to inspire and encourage the Software Testing Community to direct their attention toward SDCs as a rapidly emerging and crucial domain. The joint call is meant to help interested authors/participants reduce travel costs by selecting the most convenient and close venue.

The competition focuses on regression testing for test suites executing system tests of SDCs in simulation. Regression testing involves test selection, prioritization, and minimization. For the competition, we focus on the first aspect - the test selection, as detailed in the competition platform repository.

Link to the competition platform: https://github.com/christianbirchler-org/sdc-testing-competition

Test Selection

Test Selection is the process of picking only the relevant test cases from the test suite for a particular change. In the context of simulation-based testing for SDCs with long-running test cases, we select test cases to minimize the execution time without sacrificing testing effectiveness.

 Test Suite        Selection
[ ][ ][ ][ ]      [x][ ][x][ ]
[ ][ ][ ][ ]  ->  [x][ ][ ][x]  -> Execution of the Selection  -> Results
[ ][ ][ ][ ]      [ ][x][ ][ ]

Aim

The competition aims to get submissions of novel and cost-effective test selectors. After the submission, all tools will be evaluated by the organizers.

Upon acceptance, all participants are asked to submit a short paper about their tool to the organizers of this track. Examples of such tool reports can be found at SBFT 2024 https://sbft24.github.io/papers/)

Tool Submission

You can freely decide on the venue you want to compete in: ICST 2025 or SBFT@ICSE 2025. Participants in each venue will be evaluated and ranked independently.

The submission process is described on the competition platform.

Tool Paper Submission

The camera-ready deadline is January 10, 2024 (AoE). Please share your draft with us before 3rd January for a review.

You will be contacted by the publisher with the instructions for preparing and submitting the camera-ready version of your paper. The given page limits (i.e., 2 pages for tool papers) are strict. The list of authors (names, emails, affiliations, order) and the paper title must not change. If a correction is needed, please email us promptly.

The tool competition is built in the context of previous work. Hence, for your tool paper, consider the following references on test selection for SDCs:

@inproceedings{sdc-scissor,
title={Cost-effective simulation-based test selection in self-driving cars software with SDC-Scissor},
author={Birchler, Christian and Ganz, Nicolas and Khatiri, Sajad and Gambi, Alessio and Panichella, Sebastiano}, booktitle={2022 IEEE international conference on software analysis, evolution and reengineering (SANER)}, pages={164--168}, year={2022},
organization={IEEE}
}

@article{ml-test-selection,
title={Machine learning-based test selection for simulation-based testing of self-driving cars software},
author={Birchler, Christian and Khatiri, Sajad and Bosshard, Bill and Gambi, Alessio and Panichella, Sebastiano}, journal={Empirical Software Engineering}, volume={28}, number={3}, pages={71},
year={2023}, publisher={Springer US New York}
}

@inproceedings{teaser,
title={TEASER: Simulation-Based CAN Bus Regression Testing for Self-Driving Cars Software},
author={Birchler, Christian and Rohrbach, Cyrill and Kim, Hyeongkyun and Gambi, Alessio and Liu, Tianhai and Horneber, Jens and Kehrer, Timo and Panichella, Sebastiano},
booktitle={2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)},
pages={2058--2061}, year={2023}, organization={IEEE}
}

If submitting to ICST 2025, include the following reference:

@inproceedings{ICST-SDC2025,
author={Birchler, Christian and Klikovits, Stefan and Fazzini, Mattia and Panichella, Sebastiano},
title={{ICST} Tool Competition 2025 - Self-Driving Car Testing Track},
booktitle={{IEEE/ACM} International Conference on Software Testing, Verification and Validation, ICST 2025},
year={2025}
}

If submitting to SBFT 2025, include the following reference:

@inproceedings{SBFT-SDC2025,
  author       = {{Birchler, Christian and Klikovits, Stefan and Fazzini, Mattia and Panichella, Sebastiano},
  title        = {{SBFT} Tool Competition 2025 - CPS-SDC Regression Testing Track},
  booktitle    = {{IEEE/ACM} International Workshop on Search-Based and Fuzz Testing, SBFT@ICSE 2025},
  year         = {2025}
}

For each paper, a presentation by one of the authors at the conference is required. For registration information, please visit the ICST conference web page.

Competition Platform

All competition guidelines, descriptions, and Q&A are managed on GitHub. The competition platform is publicly available: https://github.com/christianbirchler-org/sdc-testing-competition

Participants are welcome to discuss any issues related to the competition platform directly on GitHub.

Sponsors

We thank BeamNG GmbH for their generous support for this conference and this competition track!

• BeamNG: https://beamng.tech/
• BeamNG Research: https://beamng.tech/research/

Contacts

• Christian Birchler: christian.birchler@zhaw.ch
• Stefan Klikovits: stefan.klikovits@jku.at
• Mattia Fazzini: mfazzini@umn.edu
• Sebastiano Panichella: spanichella@gmail.com