ICST 2024
Mon 27 - Fri 31 May 2024 Canada
Wed 29 May 2024 15:50 - 16:10 at Room 2 & 3 - Testing Autonomous Driving Systems Chair(s): Nargiz Humbatova

Simulation-based testing of automated driving systems (ADS) is the industry standard, being a controlled, safe, and cost-effective alternative to real-world testing. Despite these advantages, virtual simulations often fail to accurately replicate real-world conditions like image fidelity, texture representation, and environmental accuracy. This can lead to significant differences in ADS behavior between simulated and real-world domains, a phenomenon known as the sim2real gap. Researchers have used Image-to-Image (I2I) neural translation to mitigate the sim2real gap, enhancing the realism of simulated environments by transforming synthetic data into more authentic representations of real-world conditions. However, while promising, these techniques may potentially introduce artifacts, distortions, or inconsistencies in the generated data that can affect the effectiveness of ADS testing. In our empirical study, we investigated how the quality of image-to-image (I2I) techniques influences the mitigation of the sim2real gap, using a set of established metrics from the literature. We evaluated two popular generative I2I architectures, pix2pix and CycleGAN, across two ADS perception tasks at a model level, namely vehicle detection and end-to-end lane keeping, using paired simulated and real-world datasets. Our findings reveal that the effectiveness of I2I architectures varies across different ADS tasks, and existing evaluation metrics do not consistently align with the ADS behavior. Thus, we conducted task-specific fine-tuning of perception metrics, which yielded a stronger correlation. Our findings indicate that a perception metric that incorporates semantic elements, tailored to each task, can facilitate selecting the most appropriate I2I technique for a reliable assessment of the sim2real gap mitigation.

Wed 29 May

Displayed time zone: Eastern Time (US & Canada) change

15:30 - 17:00
Testing Autonomous Driving SystemsResearch Papers / Testing Tools and Demonstration at Room 2 & 3
Chair(s): Nargiz Humbatova USI Lugano
15:30
20m
Research paper
Adversarial Testing with Reinforcement Learning: A Case Study on Autonomous Driving
Research Papers
Andréa Doreste , Matteo Biagiola Università della Svizzera italiana, Paolo Tonella USI Lugano
15:50
20m
Research paper
Assessing Quality Metrics for Neural Reality Gap Input Mitigation in Autonomous Driving Testing
Research Papers
Stefano Carlo Lambertenghi Technische Universität München, fortiss GmbH, Andrea Stocco Technical University of Munich, fortiss
Pre-print
16:10
20m
Research paper
Predicting Safety Misbehaviours in Autonomous Driving Systems using Uncertainty Quantification
Research Papers
Ruben Grewal , Paolo Tonella USI Lugano, Andrea Stocco Technical University of Munich, fortiss
Pre-print
16:30
20m
Research paper
AURORA: Navigating UI Tarpits via Automated Neural Screen Understanding
Research Papers
Safwat Ali Khan George Mason University, Wenyu Wang University of Illinois Urbana-Champaign, Yiran Ren , Bin Zhu , Jiangfan Shi , Wing Lam George Mason University, Kevin Moran University of Central Florida
Pre-print
16:50
10m
Demonstration
U-Fuzz: A Tool for Stateful Fuzzing of IoT Protocols on COTS Devices
Testing Tools and Demonstration
Shang Zewen , Matheus Eduardo Garbelini , Sudipta Chattopadhyay Singapore University of Technology and Design