Smoke Testing for Machine Learning: Simple Tests to Discover Severe Defects
Tue 10 May 2022 12:00 - 12:05 at ICSE room 2-even hours - Software Testing 11 Chair(s): Jonathan Bell
Machine learning is nowadays a standard technique for data analysis within software applications. Software engineers need quality assurance techniques that are suitable for these new kinds of systems. Within this article, we discuss the question whether standard software testing techniques that have been part of textbooks since decades are also useful for the testing of machine learning software. Concretely, we try to determine generic and simple smoke tests that can be used to assert that basic functions can be executed without crashing. We found that we can derive such tests using techniques similar to equivalence classes and boundary value analysis. Moreover, we found that these concepts can also be applied to hyperparameters, to further improve the quality of the smoke tests. Even though our approach is almost trivial, we were able to find bugs in all three machine learning libraries that we tested and severe bugs in two of the three libraries. This demonstrates that common software testing techniques are still valid in the age of machine learning and that considerations how they can be adapted to this new context can help to find and prevent severe bugs, evenin mature machine learning libraries.
Tue 10 MayDisplayed time zone: Eastern Time (US & Canada) change
04:00 - 05:00 | Software Testing 1Technical Track / Journal-First Papers at ICSE room 2-even hours Chair(s): Ajitha Rajan University of Edinburgh | ||
04:00 5mTalk | The Impact of Dormant Defects on Defect Prediction: a Study of 19 Apache Projects Journal-First Papers Davide Falessi University of Rome Tor Vergata, Italy, Aalok Ahluwalia California Polytechnic State University, Massimiliano Di Penta University of Sannio, Italy Link to publication DOI Media Attached | ||
04:05 5mTalk | Smoke Testing for Machine Learning: Simple Tests to Discover Severe Defects Journal-First Papers DOI Media Attached | ||
04:10 5mTalk | RNN-Test: Towards Adversarial Testing for Recurrent Neural Network Systems Journal-First Papers Jianmin Guo Tsinghua University, Quan Zhang Tsinghua University, Yue Zhao Huawei Technologies Co., Ltd., Heyuan Shi Central South University, Yu Jiang Tsinghua University, Jia-Guang Sun Link to publication DOI Pre-print Media Attached | ||
04:15 5mTalk | Adaptive Test Selection for Deep Neural Networks Technical Track Xinyu Gao Nanjing University, Yang Feng Nanjing University, Yining Yin Nanjing University, China, Zixi Liu Nanjing University, Zhenyu Chen Nanjing University, Baowen Xu Nanjing University Pre-print Media Attached | ||
04:20 5mTalk | Evaluating and Improving Neural Program-Smoothing-based Fuzzing Technical Track Mingyuan Wu Southern University of Science and Technology, Ling Jiang Southern University of Science and Technology, Jiahong Xiang Southern University of Science and Technology, Yuqun Zhang Southern University of Science and Technology, Guowei Yang The University of Queensland, Huixin Ma Tencent Security Keen Lab, Sen Nie Keen Security Lab, Tencent, Shi Wu Tencent Security Keen Lab, Heming Cui University of Hong Kong, Lingming Zhang University of Illinois at Urbana-Champaign DOI Pre-print Media Attached | ||
04:25 5mTalk | Muffin: Testing Deep Learning Libraries via Neural Architecture Fuzzing Technical Track Jiazhen Gu Fudan University, China, Xuchuan Luo Fudan University, Yangfan Zhou Fudan University, Xin Wang Fudan University Pre-print Media Attached |