Reliability Assurance for Deep Neural Network Architectures Against Numerical Defects
With the widespread deployment of deep neural networks (DNNs), ensuring the reliability of DNN-based systems is of great importance. Serious reliability issues such as system failures can be caused by numerical defects, one of the most frequent defects in DNNs. To assure high reliability against numerical defects, in this paper, we propose the RANUM approach including novel techniques for three reliability assurance tasks: detection of potential numerical defects, confirmation of potential-defect feasibility, and suggestion of defect fixes. To the best of our knowledge, RANUM is the first approach that confirms potential-defect feasibility with failure-exhibiting tests and suggests fixes automatically. Extensive experiments on the benchmarks of 63 real-world DNN architectures show that RANUM outperforms state-of-the-art approaches across the three reliability assurance tasks. In addition, when the RANUM-generated fixes are compared with developers’ fixes on open-source projects, in 37 out of 40 cases, RANUM-generated fixes are equivalent to or even better than human fixes.
Fri 19 MayDisplayed time zone: Hobart change
11:00 - 12:30 | AI testing 2Technical Track / Journal-First Papers at Meeting Room 101 Chair(s): Gunel Jahangirova USI Lugano, Switzerland | ||
11:00 15mTalk | Aries: Efficient Testing of Deep Neural Networks via Labeling-Free Accuracy Estimation Technical Track Qiang Hu University of Luxembourg, Yuejun GUo University of Luxembourg, Xiaofei Xie Singapore Management University, Maxime Cordy University of Luxembourg, Luxembourg, Lei Ma University of Alberta, Mike Papadakis University of Luxembourg, Luxembourg, Yves Le Traon University of Luxembourg, Luxembourg Pre-print | ||
11:15 15mTalk | Testing the Plasticity of Reinforcement Learning Based Systems Journal-First Papers Link to publication DOI Pre-print | ||
11:30 15mTalk | CC: Causality-Aware Coverage Criterion for Deep Neural Networks Technical Track Zhenlan Ji The Hong Kong University of Science and Technology, Pingchuan Ma HKUST, Yuanyuan Yuan The Hong Kong University of Science and Technology, Shuai Wang Hong Kong University of Science and Technology | ||
11:45 15mTalk | Balancing Effectiveness and Flakiness of Non-Deterministic Machine Learning Tests Technical Track Chunqiu Steven Xia University of Illinois at Urbana-Champaign, Saikat Dutta University of Illinois at Urbana-Champaign, Sasa Misailovic University of Illinois at Urbana-Champaign, Darko Marinov University of Illinois at Urbana-Champaign, Lingming Zhang University of Illinois at Urbana-Champaign | ||
12:00 15mTalk | Many-Objective Reinforcement Learning for Online Testing of DNN-Enabled Systems Technical Track Fitash ul haq , Donghwan Shin The University of Sheffield, Lionel Briand University of Luxembourg; University of Ottawa Pre-print | ||
12:15 15mTalk | Reliability Assurance for Deep Neural Network Architectures Against Numerical Defects Technical Track Linyi Li University of Illinois at Urbana-Champaign, Yuhao Zhang University of Wisconsin-Madison, Luyao Ren Peking University, China, Yingfei Xiong Peking University, Tao Xie Peking University Pre-print | ||