Write a Blog >>
ICSE 2021
Mon 17 May - Sat 5 June 2021

Deep Neural Network (DNN) testing is one of the most widely-used ways to guarantee the quality of DNNs. However, labeling test inputs to check the correctness of DNN prediction is very costly, which could largely affect the efficiency of DNN testing, even the whole process of DNN development. To relieve the labeling-cost problem, we propose a novel test input prioritization approach (called PRIMA) for DNNs via intelligent mutation analysis in order to label more bug-revealing test inputs earlier for a limited time, which facilitates to improve the efficiency of DNN testing. PRIMA is based on the key insight: a test input that is able to kill many mutated models and produce different prediction results with many mutated inputs, is more likely to reveal DNN bugs, and thus it should be prioritized higher. After obtaining a number of mutation results from a series of our designed model and input mutation rules for each test input, PRIMA further incorporates learning-to-rank (a kind of supervised machine learning to solve ranking problems) to intelligently combine these mutation results for effective test input prioritization. We conducted an extensive study based on 36 popular subjects by carefully considering their diversity from five dimensions (i.e., different domains of test inputs, different DNN tasks, different network structure, different types of test inputs, and different training scenarios). Our experimental results demonstrate the effectiveness of PRIMA, significantly outperforming the state-of-the-art approaches. In particular, we have applied PRIMA to the practical autonomous-vehicle testing in a large motor company, and the results on 4 real-world scene-recognition subjects in autonomous vehicles further confirm the practicability of PRIMA.

Tue 25 May

Displayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change

12:05 - 13:05
1.2.1. Deep Neural Networks: Validation #2Technical Track at Blended Sessions Room 1 +12h
Chair(s): Grace Lewis Carnegie Mellon Software Engineering Institute
12:05
20m
Paper
Measuring Discrimination to Boost Comparative Testing for Multiple Deep Learning ModelsTechnical Track
Technical Track
Linghan Meng Nanjing University, Yanhui Li Department of Computer Science and Technology, Nanjing University, Lin Chen Department of Computer Science and Technology, Nanjing University, Zhi Wang Nanjing University, Di Wu Momenta, Yuming Zhou Nanjing University, Baowen Xu Nanjing University
Pre-print Media Attached
12:25
20m
Paper
Prioritizing Test Inputs for Deep Neural Networks via Mutation AnalysisTechnical Track
Technical Track
Zan Wang College of Intelligence and Computing, Tianjin University, Hanmo You College of Intelligence and Computing, Tianjin University, Junjie Chen College of Intelligence and Computing, Tianjin University, Yingyi Zhang College of Intelligence and Computing, Tianjin University, Xuyuan Dong Information and Network Center,Tianjin University, Wenbin Zhang Information and Network Center,Tianjin University
Pre-print Media Attached
12:45
20m
Paper
Testing Machine Translation via Referential TransparencyTechnical Track
Technical Track
Pinjia He ETH Zurich, Clara Meister ETH Zurich, Zhendong Su ETH Zurich
Pre-print Media Attached

Wed 26 May

Displayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change

00:05 - 01:05
1.2.1. Deep Neural Networks: Validation #2Technical Track at Blended Sessions Room 1
00:05
20m
Paper
Measuring Discrimination to Boost Comparative Testing for Multiple Deep Learning ModelsTechnical Track
Technical Track
Linghan Meng Nanjing University, Yanhui Li Department of Computer Science and Technology, Nanjing University, Lin Chen Department of Computer Science and Technology, Nanjing University, Zhi Wang Nanjing University, Di Wu Momenta, Yuming Zhou Nanjing University, Baowen Xu Nanjing University
Pre-print Media Attached
00:25
20m
Paper
Prioritizing Test Inputs for Deep Neural Networks via Mutation AnalysisTechnical Track
Technical Track
Zan Wang College of Intelligence and Computing, Tianjin University, Hanmo You College of Intelligence and Computing, Tianjin University, Junjie Chen College of Intelligence and Computing, Tianjin University, Yingyi Zhang College of Intelligence and Computing, Tianjin University, Xuyuan Dong Information and Network Center,Tianjin University, Wenbin Zhang Information and Network Center,Tianjin University
Pre-print Media Attached
00:45
20m
Paper
Testing Machine Translation via Referential TransparencyTechnical Track
Technical Track
Pinjia He ETH Zurich, Clara Meister ETH Zurich, Zhendong Su ETH Zurich
Pre-print Media Attached