DeepHunter: A Coverage-Guided Fuzz Testing Framework for Deep Neural Networks
The past decade has seen the great potential of applying deep neural network (DNN) based software to safety-critical scenarios, such as autonomous driving. Similar to traditional software, DNNs could exhibit incorrect behaviors, caused by hidden defects, leading to severe accidents and losses. In this paper, we propose DeepHunter, a coverage-guided fuzz testing framework for detecting potential defects of general-purpose DNNs. To this end, we first propose a new metamorphic mutation strategy to generate new semantically preserved tests, and leverage multiple extensible coverage criteria as feedback to guide the test generation. We further propose a new seed selection strategy that combines both diversity-based seed selection and recency-based seed selection. We finally implement 5 existing testing criteria and 4 seed selection strategies in DeepHunter. Large-scale experiments demonstrate that (1) the metamorphic mutation strategy is useful to generate new valid tests with the same semantics as the original seed, by a 98% validity ratio; (2) diversity-based seed selection is more important than recency-based seed selection in boosting the coverage and in detecting defects; (3) DeepHunter significantly outperforms the state of the art (TensorFuzz and DeepTest) by coverage as well as the quantity and diversity of defects identified; (4) using corner-region based criteria, DeepHunter tends to be more useful to capture defects (capture 4x more defects than TensorFuzz in MobileNet) during the DNN quantization for platform migration.
Thu 18 JulDisplayed time zone: Beijing, Chongqing, Hong Kong, Urumqi change
14:00 - 15:30 | Testing and Machine LearningTechnical Papers at Grand Ballroom Chair(s): Hongyu Zhang The University of Newcastle | ||
14:00 22mTalk | DeepHunter: A Coverage-Guided Fuzz Testing Framework for Deep Neural Networks Technical Papers Xiaofei Xie Nanyang Technological University, Lei Ma Kyushu University, Felix Juefei-Xu Carnegie Mellon University, Minhui Xue , Hongxu Chen Nanyang Technological University, Yang Liu Nanyang Technological University, Singapore, Jianjun Zhao Kyushu University, Bo Li UIUC, Jianxiong Yin NVIDIA AI Tech Centre, Simon See NVIDIA AI Tech Centre | ||
14:22 22mTalk | Search-based Test and Improvement of Machine-Learning-Based Anomaly Detection Systems Technical Papers Maxime Cordy SnT, University of Luxembourg, Steve Muller unaffiliated, Mike Papadakis University of Luxembourg, Yves Le Traon University of Luxembourg | ||
14:45 22mTalk | DeepFL: Integrating Multiple Fault Diagnosis Dimensions for Deep Fault Localization Technical Papers Xia Li University of Texas at Dallas, USA, Wei Li Southern University of Science and Technology, Yuqun Zhang Southern University of Science and Technology, Lingming Zhang | ||
15:07 22mTalk | Codebase-Adaptive Detection of Security-Relevant Methods Technical Papers Goran Piskachev Fraunhofer IEM, Lisa Nguyen Quang Do Paderborn University, Eric Bodden Heinz Nixdorf Institut, Paderborn University and Fraunhofer IEM DOI Pre-print Media Attached File Attached |