SAFE: Harnessing LLM for Scenario-Driven ADS Testing from Multimodal Crash Data
Ensuring the safety of Autonomous Driving Systems (ADS) requires realistic and reproducible test scenarios, yet extracting such scenarios from multimodal crash reports remains a major challenge. Large Language Models (LLMs) often hallucinate and lose map structure, resulting in unrealistic road layouts and vehicle behaviors. To address this, we introduce SAFE, a novel Scenario-based ADS testing Framework via multimodal Extraction, which leverages Retrieval-Augmented Generation (RAG), knowledge-grounded prompting, Chain-of-Thought (CoT) reasoning, and self-validation to improve scenario reconstruction from multimodal crash data.
SAFE achieves 93.8% accuracy in extracting road network details, 80.0% for actor information, and 100% for environmental context. In human studies, SAFE outperforms LCTGen and AC3R in reconstructing consistent road networks and vehicle behaviors. Under identical ADS and simulator settings, SAFE detects 39 and 71 more safety violations than LCTGen and AC3R, respectively, and reproduces 12 more real world crash cases than LCTGen. On 19 cases supported by AC3R, SAFE reproduces one additional crash case with statistically significant gains across five runs. It generates scenarios within 25 seconds and triggers violations after just 1 case (IDM), 3 cases (PPO), and 1 case (BeamNG). Unlike AC3R, SAFE is ontology-free and generalizes to a broader range of crash scenarios. Code: https://anonymous.4open.science/r/SAFE-8404/README.md
Wed 15 AprDisplayed time zone: Brasilia, Distrito Federal, Brazil change
16:00 - 17:30 | Testing and Analysis 6Research Track at Oceania IX Chair(s): Alberto Martin-Lopez Software Institute - USI, Lugano | ||
16:00 15mTalk | SAFE: Harnessing LLM for Scenario-Driven ADS Testing from Multimodal Crash Data Research Track Siwei Luo Macquarie University, Yang Zhang , Yao Deng Macquarie University, Linfeng Liang Macquarie University, Xi Zheng Macquarie University | ||
16:15 15mTalk | Bounded Exhaustive Random Program Generation for Testing Solidity Compilers Research Track Haoyang Ma Hong Kong University of Science and Technology, Alastair F. Donaldson Imperial College London, Qingchao Shen Tianjin University, Yongqiang Tian Monash University, Junjie Chen Tianjin University, Shing-Chi Cheung Hong Kong University of Science and Technology | ||
16:30 15mResearch paper | META²V2V: Revealing Behavioural Deviations under Mutual Perception in Multi-Vehicle Autonomous Driving Research Track Lejin Li Kyushu University, Xiao-Yi Zhang University of Science and Technology Beijing, Shuncheng Tang University of Science and Technology of China, Zhenya Zhang Kyushu University, Jianjun Zhao Kyushu University Media Attached | ||
16:45 15mTalk | DeFT: Maintaining Determinism and Extracting Unit Tests for Autonomous Driving Planning Research Track Yuqi Huai University of California, Irvine, Yuntianyi Chen University of California, Irvine, Ziwen Wan University of California, Irvine, Alfred Chen University of California, Irvine, Joshua Garcia University of California, Irvine DOI Pre-print | ||
17:00 15mTalk | TARIPlay: A Test Framework for AR Applications based on Interactive Area Detection in Playback Videos Research Track Seyed Amir Mousavi PhD Student at University of Texas at San Antonio, Xiaoyin Wang University of Texas at San Antonio | ||
17:15 15mTalk | Validating Mixed-Integer Programming Solvers Research Track Xintong Zhou University of Waterloo, Zhenyang Xu University of Waterloo, Chengnian Sun University of Waterloo | ||