Write a Blog >>
ICSE 2021
Mon 17 May - Sat 5 June 2021

Deep Neural Network (DNN) testing is one of the most widely-used ways to guarantee the quality of DNNs. However, labeling test inputs to check the correctness of DNN prediction is very costly, which could largely affect the efficiency of DNN testing, even the whole process of DNN development. To relieve the labeling-cost problem, we propose a novel test input prioritization approach (called PRIMA) for DNNs via intelligent mutation analysis in order to label more bug-revealing test inputs earlier for a limited time, which facilitates to improve the efficiency of DNN testing. PRIMA is based on the key insight: a test input that is able to kill many mutated models and produce different prediction results with many mutated inputs, is more likely to reveal DNN bugs, and thus it should be prioritized higher. After obtaining a number of mutation results from a series of our designed model and input mutation rules for each test input, PRIMA further incorporates learning-to-rank (a kind of supervised machine learning to solve ranking problems) to intelligently combine these mutation results for effective test input prioritization. We conducted an extensive study based on 36 popular subjects by carefully considering their diversity from five dimensions (i.e., different domains of test inputs, different DNN tasks, different network structure, different types of test inputs, and different training scenarios). Our experimental results demonstrate the effectiveness of PRIMA, significantly outperforming the state-of-the-art approaches. In particular, we have applied PRIMA to the practical autonomous-vehicle testing in a large motor company, and the results on 4 real-world scene-recognition subjects in autonomous vehicles further confirm the practicability of PRIMA.

Conference Day
Tue 25 May

Displayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change

12:05 - 13:05
1.2.1. Deep Neural Networks: Validation #2Technical Track at Blended Sessions Room 1 +12h
Chair(s): Grace LewisCarnegie Mellon Software Engineering Institute
12:05
20m
Paper
Measuring Discrimination to Boost Comparative Testing for Multiple Deep Learning ModelsTechnical Track
Technical Track
Linghan MengNanjing University, Yanhui LiDepartment of Computer Science and Technology, Nanjing University, Lin ChenDepartment of Computer Science and Technology, Nanjing University, Zhi WangNanjing University, Di WuMomenta, Yuming ZhouNanjing University, Baowen XuNanjing University
Pre-print Media Attached
12:25
20m
Paper
Prioritizing Test Inputs for Deep Neural Networks via Mutation AnalysisTechnical Track
Technical Track
Zan WangCollege of Intelligence and Computing, Tianjin University, Hanmo YouCollege of Intelligence and Computing, Tianjin University, Junjie ChenCollege of Intelligence and Computing, Tianjin University, Yingyi ZhangCollege of Intelligence and Computing, Tianjin University, Xuyuan DongInformation and Network Center,Tianjin University, Wenbin ZhangInformation and Network Center,Tianjin University
Pre-print Media Attached
12:45
20m
Paper
Testing Machine Translation via Referential TransparencyTechnical Track
Technical Track
Pinjia HeETH Zurich, Clara MeisterETH Zurich, Zhendong SuETH Zurich
Pre-print Media Attached

Conference Day
Wed 26 May

Displayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change

00:05 - 01:05
1.2.1. Deep Neural Networks: Validation #2Technical Track at Blended Sessions Room 1
00:05
20m
Paper
Measuring Discrimination to Boost Comparative Testing for Multiple Deep Learning ModelsTechnical Track
Technical Track
Linghan MengNanjing University, Yanhui LiDepartment of Computer Science and Technology, Nanjing University, Lin ChenDepartment of Computer Science and Technology, Nanjing University, Zhi WangNanjing University, Di WuMomenta, Yuming ZhouNanjing University, Baowen XuNanjing University
Pre-print Media Attached
00:25
20m
Paper
Prioritizing Test Inputs for Deep Neural Networks via Mutation AnalysisTechnical Track
Technical Track
Zan WangCollege of Intelligence and Computing, Tianjin University, Hanmo YouCollege of Intelligence and Computing, Tianjin University, Junjie ChenCollege of Intelligence and Computing, Tianjin University, Yingyi ZhangCollege of Intelligence and Computing, Tianjin University, Xuyuan DongInformation and Network Center,Tianjin University, Wenbin ZhangInformation and Network Center,Tianjin University
Pre-print Media Attached
00:45
20m
Paper
Testing Machine Translation via Referential TransparencyTechnical Track
Technical Track
Pinjia HeETH Zurich, Clara MeisterETH Zurich, Zhendong SuETH Zurich
Pre-print Media Attached