Write a Blog >>
ASE 2020
Mon 21 - Fri 25 September 2020 Melbourne, Australia
Wed 23 Sep 2020 01:10 - 01:30 at Kangaroo - Software Engineering for AI (2) Chair(s): Aldeida Aleti

Deep learning (DL) has been applied widely, and the quality of DL system becomes crucial, especially for safety-critical applications. Existing work mainly focuses on the quality analysis of DL models, but lacks attention to the underlying libraries and frameworks on which all DL models depend. In this work, we propose Audee, anovel approach for testing DL libraries and localizing bugs. Audee adopts a search-based approach and implements three different mutation strategies to generate diverse tests cases by exploring combinations of model structures, parameters, weights and inputs. Audee is able to detect three types of bugs: logic bugs, crashes and Not-a-Number (NaN) bugs. In particular, for logic bugs, Audee adopts a cross-reference check to detect behavioral inconsistencies across multiple frameworks (e.g., TensorFlow and PyTorch), which indicates potential bugs in their implementations. For NaN bugs, Audee adopts a heuristic-based approach to generate DNNs that tend to output outliers (i.e., too large or small values), and these values are likely to cause NaN value. Furthermore, Audee leverages causal testing based technique to localize layers as well as parameters that cause inconsistencies or bugs. To evaluate the effectiveness of our approach, we applied Audeeon evaluating four DL frameworks, i.e., TensorFlow, CNTK, Theano, and PyTorch. We totally generate 260 models which cover 25 widely-used APIs in the four frameworks. The results demonstrate Audee are effective indetecting inconsistencies, crashes and NaN bugs. In total, 26 unique unknown bugs were discovered, and seven of them have already been confirmed by the developers.

Wed 23 Sep
Times are displayed in time zone: (UTC) Coordinated Universal Time change

01:10 - 02:10: Software Engineering for AI (2) Research Papers / Industry Showcase at Kangaroo
Chair(s): Aldeida AletiMonash University
01:10 - 01:30
Audee: Automated Testing for Deep Learning Frameworks
Research Papers
Qianyu GuoCollege of Intelligence and Computing, Tianjin University, Xiaofei XieNanyang Technological University, Yi LiNanyang Technological University, Singapore, Xiaoyu ZhangXi'an Jiaotong University, Yang LiuNanyang Technological University, Singapore, Li XiaohongTianJin University, Chao ShenXi'an Jiaotong University
01:30 - 01:50
Towards Interpreting Recurrent Neural Networks through Probabilistic Abstraction
Research Papers
Guoliang DongComputer College of Zhejiang University, Jingyi WangZhejiang University, Jun SunSingapore Management University, Yang ZhangZhejiang University, Xinyu WangZhejiang University, Dai TingHuawei International Pte Ltd, Jin Song DongNational University of Singapore, Xingen WangZhejiang University
01:50 - 02:10
Towards Building Robust DNN Applications: An Industrial Case Study of Evolutionary Data Augmentation
Industry Showcase
Haruki YokoyamaFujitsu Laboratories Ltd., Satoshi OnoueFujitsu Ltd., Shinji KikuchiFujitsu Laboratories Ltd.