DiverGet: a Search-Based Software Testing approach for Deep Neural Network Quantization assessment
Quantization is one of the most applied Deep Neural Network (DNN) compression strategies, when deploying a trained DNN model on an embedded system or a cell phone. This is owing to its simplicity and adaptability to a wide range of applications and circumstances, as opposed to specific Artificial Intelligence (AI) accelerators and compilers that are often designed only for certain specific hardware (e.g., Google Coral Edge TPU). With the growing demand for quantization, ensuring the reliability of this strategy is becoming a critical challenge. Traditional testing methods, which gather more and more genuine data for better assessment, are often not practical because of the large size of the input space and the high similarity between the original DNN and its quantized counterpart. As a result, advanced assessment strategies have become of paramount importance. In this paper, we present DiverGet, a search based testing framework for quantization assessment. DiverGet defines a space of metamorphic relations that simulate naturally-occurring distortions on the inputs. Then, it optimally explores these relations to reveal the disagreements among DNNs of different arithmetic precision. We evaluate the performance of DiverGet on state-of-the-art DNNs applied to hyperspectral remote sensing images. We chose the remote sensing DNNs as they’re being increasingly deployed at the edge (e.g., high-lift drones) in critical domains like climate change research and astronomy. Our results show that DiverGet successfully challenges the robustness of established quantization techniques against naturally-occurring shifted data, and outperforms its most recent concurrent, DiffChaser, with a success rate that is (on average) four times higher.
Thu 18 MayDisplayed time zone: Hobart change
11:00 - 12:30 | AI testing 1Technical Track / DEMO - Demonstrations / Journal-First Papers at Meeting Room 102 Chair(s): Matthew B Dwyer University of Virginia | ||
11:00 15mTalk | When and Why Test Generators for Deep Learning Produce Invalid Inputs: an Empirical Study Technical Track Pre-print | ||
11:15 15mTalk | Fuzzing Automatic Differentiation in Deep-Learning Libraries Technical Track Chenyuan Yang University of Illinois at Urbana-Champaign, Yinlin Deng University of Illinois at Urbana-Champaign, Jiayi Yao The Chinese University of Hong Kong, Shenzhen, Yuxing Tu Huazhong University of Science and Technology, Hanchi Li University of Science and Technology of China, Lingming Zhang University of Illinois at Urbana-Champaign | ||
11:30 15mTalk | Lightweight Approaches to DNN Regression Error Reduction: An Uncertainty Alignment Perspective Technical Track Zenan Li Nanjing University, China, Maorun Zhang Nanjing University, China, Jingwei Xu , Yuan Yao Nanjing University, Chun Cao Nanjing University, Taolue Chen Birkbeck University of London, Xiaoxing Ma Nanjing University, Jian Lu Nanjing University Pre-print | ||
11:45 7mTalk | DeepJudge: A Testing Framework for Copyright Protection of Deep Learning Models DEMO - Demonstrations Jialuo Chen Zhejiang University, Youcheng Sun The University of Manchester, Jingyi Wang Zhejiang University, Peng Cheng Zhejiang University, Xingjun Ma Deakin University | ||
11:52 7mTalk | DeepCrime: from Real Faults to Mutation Testing Tool for Deep Learning DEMO - Demonstrations | ||
12:00 7mTalk | DiverGet: a Search-Based Software Testing approach for Deep Neural Network Quantization assessment Journal-First Papers Ahmed Haj Yahmed École Polytechnique de Montréal, Houssem Ben Braiek École Polytechnique de Montréal, Foutse Khomh Polytechnique Montréal, Sonia Bouzidi National Institute of Applied Science and Technology, Rania Zaatour Potsdam Institute for Climate Impact Research | ||
12:07 15mTalk | Revisiting Neuron Coverage for DNN Testing: A Layer-Wise and Distribution-Aware Criterion Technical Track Yuanyuan Yuan The Hong Kong University of Science and Technology, Qi Pang HKUST, Shuai Wang Hong Kong University of Science and Technology |