Deep learning (DL) has attracted wide attention and has been widely deployed in recent years. As a result, more and more research efforts have been dedicated to testing DL libraries and frameworks. However, existing work largely overlooks one crucial component of any DL system, automatic differentiation (AD), which is the basis for the recent development of DL. To this end, we propose $\nabla$Fuzz, the first general and practical approach specifically targeting the critical AD component in DL libraries. Our key insight is that each DL library API can be abstracted into a function processing tensors/vectors, which can be differentially tested under various execution scenarios (for computing outputs/gradients with different implementations). We have implemented $\nabla$Fuzz as a fully automated API-level fuzzer targeting AD in DL libraries, which utilizes differential testing on different execution scenarios to test both first-order and high-order gradients, and also includes automated filtering strategies to remove false positives caused by numerical instability. We have performed an extensive study on four of the most popular and actively-maintained DL libraries, PyTorch, TensorFlow, JAX, and OneFlow. The result shows that $\nabla$Fuzz substantially outperforms state-of-the-art fuzzers in terms of both code coverage and bug detection. To date, $\nabla$Fuzz has detected 173 bugs for the studied DL libraries, with 144 already confirmed by developers (117 of which are previously unknown bugs and 107 are related to AD). None of the confirmed AD bugs were detected by existing fuzzers.
Thu 18 MayDisplayed time zone: Hobart change
11:00 - 12:30 | AI testing 1Technical Track / DEMO - Demonstrations / Journal-First Papers at Meeting Room 102 Chair(s): Matthew B Dwyer University of Virginia | ||
11:00 15mTalk | When and Why Test Generators for Deep Learning Produce Invalid Inputs: an Empirical Study Technical Track Pre-print | ||
11:15 15mTalk | Fuzzing Automatic Differentiation in Deep-Learning Libraries Technical Track Chenyuan Yang University of Illinois at Urbana-Champaign, Yinlin Deng University of Illinois at Urbana-Champaign, Jiayi Yao The Chinese University of Hong Kong, Shenzhen, Yuxing Tu Huazhong University of Science and Technology, Hanchi Li University of Science and Technology of China, Lingming Zhang University of Illinois at Urbana-Champaign | ||
11:30 15mTalk | Lightweight Approaches to DNN Regression Error Reduction: An Uncertainty Alignment Perspective Technical Track Zenan Li Nanjing University, China, Maorun Zhang Nanjing University, China, Jingwei Xu , Yuan Yao Nanjing University, Chun Cao Nanjing University, Taolue Chen Birkbeck University of London, Xiaoxing Ma Nanjing University, Jian Lu Nanjing University Pre-print | ||
11:45 7mTalk | DeepJudge: A Testing Framework for Copyright Protection of Deep Learning Models DEMO - Demonstrations Jialuo Chen Zhejiang University, Youcheng Sun The University of Manchester, Jingyi Wang Zhejiang University, Peng Cheng Zhejiang University, Xingjun Ma Deakin University | ||
11:52 7mTalk | DeepCrime: from Real Faults to Mutation Testing Tool for Deep Learning DEMO - Demonstrations | ||
12:00 7mTalk | DiverGet: a Search-Based Software Testing approach for Deep Neural Network Quantization assessment Journal-First Papers Ahmed Haj Yahmed École Polytechnique de Montréal, Houssem Ben Braiek École Polytechnique de Montréal, Foutse Khomh Polytechnique Montréal, Sonia Bouzidi National Institute of Applied Science and Technology, Rania Zaatour Potsdam Institute for Climate Impact Research | ||
12:07 15mTalk | Revisiting Neuron Coverage for DNN Testing: A Layer-Wise and Distribution-Aware Criterion Technical Track Yuanyuan Yuan The Hong Kong University of Science and Technology, Qi Pang HKUST, Shuai Wang Hong Kong University of Science and Technology | ||