Predoo: Precision Testing of Deep Learning Operators
Fri 16 Jul 2021 11:00 - 11:20 at ISSTA 1 - Session 17 (time band 3) Testing Miscellaneous Systems Chair(s): Thuan Pham
Deep learning(DL) techniques attract people from various fields with superior performance in making progressive breakthroughs. To ensure the quality of DL techniques, researchers have been working on testing and verification approaches. Some recent studies reveal that the underlying DL operators could cause defects inside a DL model. DL operators work as fundamental components in DL libraries. Library developers still work on practical approaches to ensure the quality of operators they provide. However, the variety of DL operators and the implementation complexity make it challenging to evaluate their quality. Operator testing with limited test cases may fail to reveal hidden defects inside the implementation. Besides, the existing model-to-library testing approach requires extra labor and time cost to identify and locate errors, i.e., developers can only react to the exposed defects. This paper proposes a fuzzing-based operator-level precision testing approach to estimate individual DL operators' precision errors to bridge this gap. Unlike conventional fuzzing techniques, valid shape variable inputs and fine-grained precision error evaluation are implemented. The testing of DL operators is treated as a searching problem to maximize output precision errors. We implement our approach in a tool named Predoo and conduct an experiment on seven DL operators from TensorFlow. The experiment result shows that Predoo can trigger larger precision errors compared to the error threshold declared in the testing scripts from the TensorFlow repository.
Thu 15 JulDisplayed time zone: Brussels, Copenhagen, Madrid, Paris change
19:00 - 20:20 | Session 11 (time band 1) Machine Learning and TestingTechnical Papers at ISSTA 1 Chair(s): August Shi University of Texas at Austin | ||
19:00 20mTalk | Interval Constraint-Based Mutation Testing of Numerical Specifications Technical Papers Clothilde Jeangoudoux MPI-SWS, Eva Darulova MPI-SWS, Christoph Lauter University of Alaska at Anchorage DOI | ||
19:20 20mTalk | Predoo: Precision Testing of Deep Learning Operators Technical Papers Xufan Zhang Nanjing University, Ning Sun Nanjing University, Chunrong Fang Nanjing University, Jiawei Liu Nanjing University, Jia Liu Nanjing University, Dong Chai Huawei, Jiang Wang Huawei, Zhenyu Chen Nanjing University DOI | ||
19:40 20mTalk | TERA: Optimizing Stochastic Regression Tests in Machine Learning Projects Technical Papers Saikat Dutta University of Illinois at Urbana-Champaign, Jeeva Selvam University of Illinois at Urbana-Champaign, Aryaman Jain University of Illinois at Urbana-Champaign, Sasa Misailovic University of Illinois at Urbana-Champaign DOI | ||
20:00 20mTalk | Deep Just-in-Time Defect Prediction: How Far Are We? Technical Papers Zhengran Zeng Southern University of Science and Technology, Yuqun Zhang Southern University of Science and Technology, Haotian Zhang Kwai, Lingming Zhang University of Illinois at Urbana-Champaign DOI |
Fri 16 JulDisplayed time zone: Brussels, Copenhagen, Madrid, Paris change
10:00 - 11:20 | Session 17 (time band 3) Testing Miscellaneous Systems Technical Papers at ISSTA 1 Chair(s): Thuan Pham The University of Melbourne | ||
10:00 20mTalk | Empirical Evaluation of Smart Contract Testing: What Is the Best Choice? Technical Papers Meng Ren Tsinghua University, Zijing Yin Tsinghua University, Fuchen Ma Tsinghua University, Zhenyang Xu University of Waterloo, Yu Jiang Tsinghua University, Chengnian Sun University of Waterloo, Huizhong Li WeBank, Yan Cai Institute of Software at Chinese Academy of Sciences DOI File Attached | ||
10:20 20mTalk | Interval Constraint-Based Mutation Testing of Numerical Specifications Technical Papers Clothilde Jeangoudoux MPI-SWS, Eva Darulova MPI-SWS, Christoph Lauter University of Alaska at Anchorage DOI | ||
10:40 20mTalk | TERA: Optimizing Stochastic Regression Tests in Machine Learning Projects Technical Papers Saikat Dutta University of Illinois at Urbana-Champaign, Jeeva Selvam University of Illinois at Urbana-Champaign, Aryaman Jain University of Illinois at Urbana-Champaign, Sasa Misailovic University of Illinois at Urbana-Champaign DOI | ||
11:00 20mTalk | Predoo: Precision Testing of Deep Learning Operators Technical Papers Xufan Zhang Nanjing University, Ning Sun Nanjing University, Chunrong Fang Nanjing University, Jiawei Liu Nanjing University, Jia Liu Nanjing University, Dong Chai Huawei, Jiang Wang Huawei, Zhenyu Chen Nanjing University DOI |