Simple Techniques Work Surprisingly Well for Neural Network Test Prioritization and Active Learning (Replicability Study)
Fri 22 Jul 2022 15:40 - 16:00 at ISSTA 2 - Session 3-10: Neural Networks, Learning, NLP C
Test Input Prioritizers (TIP) for Deep Neural Networks (DNN) are an important technique to handle the typically very large test datasets efficiently, saving computation and labelling costs. This is particularly true for large scale, deployed systems, where inputs observed in production are recorded to serve as potential test or training data for next versions of the system. Feng et. al. propose DeepGini, a very fast and simple TIP and show that it outperforms more elaborate techniques such as neuron- and surprise coverage. In a large-scale study (4 case studies, 8 test datasets, 32’200 trained models) we verify their findings. However, we also find that other comparable or even simpler baselines from the field of uncertainty quantification, such as the predicted softmax likelihood or the entropy of the predicted softmax likelihoods perform equally well as DeepGini.
Fri 22 JulDisplayed time zone: Seoul change
00:00 - 01:00 | |||
00:00 20mTalk | AEON: A Method for Automatic Evaluation of NLP Test Cases Technical Papers Jen-tse Huang The Chinese University of Hong Kong, Jianping Zhang The Chinese University of Hong Kong, Wenxuan Wang The Chinese University of Hong Kong, Pinjia He The Chinese University of Hong Kong, Shenzhen, Yuxin Su Sun Yat-sen University, Michael Lyu The Chinese University of Hong Kong DOI | ||
00:20 20mTalk | Simple Techniques Work Surprisingly Well for Neural Network Test Prioritization and Active Learning (Replicability Study) Technical Papers DOI Pre-print | ||
00:40 20mTalk | ε-weakened Robustness of Deep Neural Networks Technical Papers Pei Huang State Key Laboratory of Computer Science, Institution of Software, Chinese Academy of Sciences, Yuting Yang Institute of Computing Technology,Chinese Academy of Sciences; University of Chinese Academy of Sciences, Minghao Liu Institute of Software, Chinese Academy of Sciences, Fuqi Jia State Key Laboratory of Computer Science, Institution of Software, Chinese Academy of Sciences, Feifei Ma Institute of Software, Chinese Academy of Sciences, Jian Zhang Institute of Software at Chinese Academy of Sciences, China DOI |
15:00 - 16:20 | |||
15:00 20mTalk | ASRTest: Automated Testing for Deep-Neural-Network-Driven Speech Recognition Systems Technical Papers Pin Ji Nanjing University, Yang Feng Nanjing University, Jia Liu Nanjing University, Zhihong Zhao Nanjing Tech Unniversity, Zhenyu Chen Nanjing University DOI | ||
15:20 20mTalk | BET: Black-box Efficient Testing for Convolutional Neural Networks Technical Papers Wang Jialai Tsinghua University, Han Qiu Tsinghua University, Yi Rong Tsinghua University, Hengkai Ye Purdue University, Qi Li Tsinghua University, Zongpeng Li Tsinghua University, Chao Zhang Tsinghua University DOI | ||
15:40 20mTalk | Simple Techniques Work Surprisingly Well for Neural Network Test Prioritization and Active Learning (Replicability Study) Technical Papers DOI Pre-print | ||
16:00 20mTalk | ε-weakened Robustness of Deep Neural Networks Technical Papers Pei Huang State Key Laboratory of Computer Science, Institution of Software, Chinese Academy of Sciences, Yuting Yang Institute of Computing Technology,Chinese Academy of Sciences; University of Chinese Academy of Sciences, Minghao Liu Institute of Software, Chinese Academy of Sciences, Fuqi Jia State Key Laboratory of Computer Science, Institution of Software, Chinese Academy of Sciences, Feifei Ma Institute of Software, Chinese Academy of Sciences, Jian Zhang Institute of Software at Chinese Academy of Sciences, China DOI |