Generating Latent Space-Aware Test Cases for Neural Networks Using Gradient-Based Search
Autonomous vehicles rely on deep learning (DL) models like object detectors and traffic sign classifiers. Assessing the robustness of these safety-critical components requires good test cases that are both realistic, lying in the distribution of the real-world data, and cost-effective in revealing potential failures. Unlike previous methods that use adversarial attacks on the pixel space, our approach identifies latent space-aware test cases using a conditional variational autoencoder (CVAE) through three steps: (1) Train a CVAE on the dataset. (2) Generate test cases by computing adversarial examples in the CVAE’s latent space. (3) Cluster challenging test cases based on their latent representations. The resulting clusters characterize regions that reveal potential defects in the DL model, which require further analysis. Our results show that our approach is capable of generating failing test cases for all classes of the MNIST and GTSRB datasets in a purely data-driven way, surpassing the baseline of random latent space sampling by up to 75 times. Finally, we validate our approach by detecting previously introduced faults in a faulty DL model. We suggest complementing expert-driven testing methods with our purely data-driven approach to uncover defects experts otherwise might miss. To strengthen transparency and facilitate replication, we provide a replication package and digital appendix to make our code, models, visualizations, and results publicly available at https://figshare.com/s/2dd8f25fb7090ba297a7.
Tue 1 AprDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
16:00 - 17:20 | |||
16:00 30mTalk | Generating Latent Space-Aware Test Cases for Neural Networks Using Gradient-Based Search AIST Simon Speth Technical University of Munich, Christoph Jasper TUM, Claudius Jordan , Alexander Pretschner TU Munich Pre-print | ||
16:30 20mTalk | Test2Text: AI-Based Mapping between Autogenerated Tests and Atomic Requirements AIST Elena Treshcheva Exactpro, Iosif Itkin Exactpro Systems, Rostislav Yavorskiy Exactpro Systems, A: Nikolai Dorofeev | ||
16:50 30mTalk | LLM Prompt Engineering for Automated White-Box Integration Test Generation in REST APIs (pre-recorded video presentation + online Q&A) AIST André Mesquita Rincon Federal Institute of Tocantins (IFTO) / Federal University of São Carlos (UFSCar), Auri Vincenzi Federal University of São Carlos, João Pascoal Faria Faculty of Engineering, University of Porto and INESC TEC |