SCTrans: Constructing a Large Public Scenario Dataset for Simulation Testing of Autonomous Driving Systems
For the safety assessment of autonomous driving systems (ADS), simulation testing has become an important complementary technique to physical road testing. In essence, simulation testing is a scenario-driven approach, whose effectiveness is highly dependent on the quality of given simulation scenarios. Moreover, simulation scenarios should be encoded into well-formatted files, otherwise, ADS simulation platforms cannot take them as inputs. Without large public datasets of simulation scenario files, both industry and academic applications of ADS simulation testing are hindered.
To fill this gap, we propose a transformation-based approach SCTrans to construct simulation scenario files, utilizing existing traffic scenario datasets (i.e., naturalistic movement of road users recorded on public roads) as data sources. Specifically, we try to transform existing traffic scenario recording files into simulation scenario files that are compatible with the most advanced ADS simulation platforms, and this task is formalized as a Model Transformation Problem. Following this idea, we construct a dataset consisting of over 1,900 diverse simulation scenarios, each of which can be directly used to test the state-of-the-art ADSs (i.e., Apollo and Autoware) via high-fidelity simulators (i.e., Carla and LGSVL). To further demonstrate the utility of our dataset, we showcase that it can boost the collision-finding capability of existing simulation-based ADS fuzzers, helping identify about 7 times more unique ADS-involved collisions within the same time period. By analyzing these collisions at the code level, we identify 9 safety-critical bugs of Apollo and Autoware, each of which can be stably exploited to cause vehicle crashes. Till now, 4 of them have been confirmed.
Thu 18 AprDisplayed time zone: Lisbon change
14:00 - 15:30 | Testing 4Research Track / Journal-first Papers at Grande Auditório Chair(s): Shiva Nejati University of Ottawa | ||
14:00 15mTalk | Concrete Constraint Guided Symbolic Execution Research Track Yue Sun Institute of Information Engineering, CAS, China, Guowei Yang University of Queensland, Shichao Lv College of Cyberspace Security, Chinese Academy of Sciences, Zhi Li Institute of Information Engineering, Chinese Academy of Sciences, China, Limin Sun Institute of Information Engineering, Chinese Academy of Sciences, School of Cyber Security, University of Chinese Academy of Sciences, Pre-print | ||
14:15 15mTalk | Improving Testing Behavior by Gamifying IntelliJ Research Track DOI Pre-print | ||
14:30 15mTalk | SCTrans: Constructing a Large Public Scenario Dataset for Simulation Testing of Autonomous Driving Systems Research Track Jiarun Dai Fudan University, Bufan Gao Fudan University, Mingyuan Luo Fudan University, Zongan Huang Fudan University, Zhongrui Li Fudan University, Yuan Zhang Fudan University, Min Yang Fudan University | ||
14:45 15mTalk | Curiosity-Driven Testing for Sequential Decision-Making Process Research Track Junda He Singapore Management University, Zhou Yang Singapore Management University, Jieke Shi Singapore Management University, Chengran Yang Singapore Management University, Singapore, Kisub Kim Singapore Management University, Singapore, Bowen Xu North Carolina State University, Xin Zhou Singapore Management University, Singapore, David Lo Singapore Management University | ||
15:00 15mTalk | Detecting Logic Bugs in Graph Database Management Systems via Injective and Surjective Graph Pattern Transformation Research Track Yuancheng Jiang National University of Singapore, Jiahao Liu National University of Singapore, Jinsheng Ba National University of Singapore, Roland H. C. Yap National University of Singapore, Singapore, Zhenkai Liang National University of Singapore, Manuel Rigger National University of Singapore DOI Pre-print | ||
15:15 7mTalk | Testing Causality in Scientific Modelling Software Journal-first Papers Andrew Graham Clark The University of Sheffield, Michael Foster The University of Sheffield, Neil Walkinshaw The University of Sheffield, Robert Hierons The University of Sheffield, Benedikt Prifling Ulm University, Volker Schmidt Ulm University, Robert D. Turner The University of Sheffield | ||
15:22 7mTalk | HybridCISave: A Combined Build and Test Selection Approach in Continuous Integration Journal-first Papers |