Write a Blog >>
ICSE 2021
Mon 17 May - Sat 5 June 2021

With the wide use of Deep Learning (DL) systems, academy and industry begin to pay attention to their quality. Testing is one of the major methods of quality assurance. However, existing testing techniques focus on the quality of DL models but lacks attention to the core underlying inference engines (i.e., frameworks and libraries). Inspired by the success stories of fuzz testing, we design a graph-based fuzz testing method to improve the quality of DL inference engines. This method is naturally followed by the graph structure of DL models. An operator-level coverage based on graph theory is introduced and six different mutations are implemented to generate diversified DL models by exploring combinations of model structures, parameters, and data. The Monte Carlo Tree Search (MCTS) is used to drive DL model generation without a training process. The experimental results show that the MCTS outperforms the random method in boosting operator-level coverage and detecting exceptions. Our method has discovered more than 40 different exceptions in three types of undesired behaviors: model conversion failure, inference failure, output comparison failure. The mutation strategies are useful to generate new valid test inputs, by up to an 8.2% more operator-level coverage on average and 8.6 more exceptions captured.

Graph-based fuzz testing for deep learning inference engines (video) (presentation_compressed.mp4)18.14MiB

Wed 26 May

Displayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change

11:20 - 12:20
2.1.2. Deep Neural Networks: Quality AssuranceTechnical Track at Blended Sessions Room 2 +12h
Chair(s): Gregorio Robles Universidad Rey Juan Carlos
11:20
20m
Paper
Graph-based Fuzz Testing for Deep Learning Inference EnginesTechnical Track
Technical Track
Weisi Luo I&V Dept of Kirin Solution Dept, HS, Huawei, Xiaoyue Run I&V Dept of Kirin Solution Dept, HS, Huawei, Dong Chai I&V Dept of Kirin Solution Dept, HS, Huawei, Jiang Wang I&V Dept of Kirin Solution Dept, HS, Huawei, Chunrong Fang Nanjing University, Zhenyu Chen Nanjing University
Pre-print Media Attached File Attached
11:40
20m
Paper
RobOT: Robustness-Oriented Testing for Deep Learning SystemsTechnical Track
Technical Track
Jingyi Wang Zhejiang University, Jialuo Chen Zhejiang University, Youcheng Sun Queen's University Belfast, UK, Xingjun Ma Deakin University, Dongxia Wang Zhejiang University, Jun Sun Singapore Management University, Singapore, Peng Cheng Zhejiang University
Pre-print Media Attached
12:00
20m
Paper
Scalable Quantitative Verification For Deep Neural NetworksArtifact ReusableTechnical Track
Technical Track
Teodora Baluta National University of Singapore, Zheng Leong Chua Independent Researcher, Kuldeep S. Meel National University of Singapore, Prateek Saxena National University of Singapore
Pre-print Media Attached
23:20 - 00:20
2.1.2. Deep Neural Networks: Quality AssuranceTechnical Track at Blended Sessions Room 2
23:20
20m
Paper
Graph-based Fuzz Testing for Deep Learning Inference EnginesTechnical Track
Technical Track
Weisi Luo I&V Dept of Kirin Solution Dept, HS, Huawei, Xiaoyue Run I&V Dept of Kirin Solution Dept, HS, Huawei, Dong Chai I&V Dept of Kirin Solution Dept, HS, Huawei, Jiang Wang I&V Dept of Kirin Solution Dept, HS, Huawei, Chunrong Fang Nanjing University, Zhenyu Chen Nanjing University
Pre-print Media Attached File Attached
23:40
20m
Paper
RobOT: Robustness-Oriented Testing for Deep Learning SystemsTechnical Track
Technical Track
Jingyi Wang Zhejiang University, Jialuo Chen Zhejiang University, Youcheng Sun Queen's University Belfast, UK, Xingjun Ma Deakin University, Dongxia Wang Zhejiang University, Jun Sun Singapore Management University, Singapore, Peng Cheng Zhejiang University
Pre-print Media Attached
00:00
20m
Paper
Scalable Quantitative Verification For Deep Neural NetworksArtifact ReusableTechnical Track
Technical Track
Teodora Baluta National University of Singapore, Zheng Leong Chua Independent Researcher, Kuldeep S. Meel National University of Singapore, Prateek Saxena National University of Singapore
Pre-print Media Attached