Bridging the Gap between Real-world and Synthetic Images for Testing Autonomous Driving Systems
Deep Neural Networks (DNNs) for Autonomous Driving Systems (ADS) are typically trained on real-world images and tested using synthetic images from simulators. This approach results in training and test datasets with dissimilar distributions, which can potentially lead to erroneously decreased test accuracy. To address this issue, the deep learning literature suggests applying domain-to-domain translators to test datasets to bring them closer to the training datasets. However, translating images used for testing may unpredictably affect the reliability, effectiveness and efficiency of the testing process. Hence, this paper investigates the following questions in the context of ADS: Could translators reduce the effectiveness of images used for ADS-DNN testing and their ability to reveal faults in ADS-DNNs? Can translators result in excessive time overhead during simulation-based testing? To address these questions, we consider three domain-to-domain translators: CycleGAN and neural style transfer, both widely used in the literature, and SAEVAE, our proposed translator. Our results for two critical ADS tasks – lane keeping and object detection – indicate that translators significantly narrow the gap in ADS test accuracy caused by distribution dissimilarities between training and test data, with SAEVAE outperforming the other two translators. We show that, based on the recent diversity, coverage, and fault-revealing ability metrics for testing deep-learning systems, translators do not compromise the diversity and the coverage of test data nor do they lead to revealing fewer faults in ADS-DNNs. Furthermore, among the three translators considered, SAEVAE incurs a negligible overhead in simulation time and can be efficiently integrated into simulation-based testing. Finally, we observe that translators increase the correlation between offline and simulation-based testing results, which can help reduce the cost of simulation-based testing.
Wed 30 OctDisplayed time zone: Pacific Time (US & Canada) change
13:30 - 15:00 | Autonomous SystemsResearch Papers / Journal-first Papers / Industry Showcase at Gardenia Chair(s): Qingkai Shi Nanjing University | ||
13:30 15mTalk | SoVAR: Build Generalizable Scenarios from Accident Reports for Autonomous Driving Testing Research Papers An Guo Nanjing University, Yuan Zhou Nanyang Technological University, Haoxiang Tian Nanyang Technological University, Chunrong Fang Nanjing University, Yunjian Sun Nanjing University, Weisong Sun Nanyang Technological University, Xinyu Gao , Luu Anh Tuan Nanyang Technological University, Yang Liu Nanyang Technological University, Zhenyu Chen Nanjing University Pre-print | ||
13:45 15mTalk | Bridging the Gap between Real-world and Synthetic Images for Testing Autonomous Driving Systems Research Papers | ||
14:00 15mTalk | In-Simulation Testing of Deep Learning Vision Models in Autonomous Robotic Manipulators Industry Showcase Dmytro Humeniuk Polytechnique Montréal, Houssem Ben Braiek Sycodal, Thomas Reid Sycodal, Foutse Khomh Polytechnique Montréal | ||
14:15 15mTalk | LeGEND: A Top-Down Approach to Scenario Generation of Autonomous Driving Systems Assisted by Large Language Models Research Papers Shuncheng Tang University of Science and Technology of China, Zhenya Zhang Kyushu University, Japan, Jixiang Zhou University of Science and Technology of China, Lei Wang National University of Defense Technology, Yuan Zhou Zhejiang Sci-Tech University, Yinxing Xue University of Science and Technology of China | ||
14:30 15mTalk | ROCAS: Root Cause Analysis of Autonomous Driving Accidents via Cyber-Physical Co-mutation Research Papers Shiwei Feng Purdue University, Yapeng Ye Purdue University, Qingkai Shi Nanjing University, Zhiyuan Cheng Purdue University, Xiangzhe Xu Purdue University, Siyuan Cheng Purdue University, Hongjun Choi DGIST, Xiangyu Zhang Purdue University | ||
14:45 15mTalk | The IDEA of Us: An Identity-Aware Architecture for Autonomous Systems Journal-first Papers Carlos Gavidia-Calderon The Alan Turing Institute, Anastasia Kordoni Lancaster University (UK), Amel Bennaceur The Open University, UK, Mark Levine Lancaster University, Bashar Nuseibeh The Open University, UK; Lero, University of Limerick, Ireland |