Write a Blog >>
ICSE 2021
Mon 17 May - Sat 5 June 2021

Recently, there has been a significant growth of interest in applying software engineering techniques for the quality assurance of deep learning (DL) systems. One popular direction is deep learning testing, where adversarial examples (a.k.a.~bugs) of DL systems are found either by fuzzing or guided search with the help of certain testing metrics. However, recent studies have revealed that the commonly used neuron coverage metrics by existing DL testing approaches are not correlated to model robustness. It is also not an effective measurement on the confidence of the model robustness after testing. In this work, we address this gap by proposing a novel testing framework called \emph{Rob}ustness-\emph{O}riented \emph{T}esting (RobOT). A key part of RobOT is a quantitative measurement on 1) the value of each test case in improving model robustness (often via retraining), and 2) the convergence quality of the model robustness improvement. RobOT utilizes the proposed metrics to automatically generate test cases valuable for improving model robustness. The proposed metrics are also a strong indicator on how well robustness improvement has converged through testing. Experiments on multiple benchmark datasets confirm the effectiveness and efficiency of RobOT in improving DL model robustness, with 67.02% increase on the adversarial robustness that is 50.65% higher than the state-of-the-art work DeepGini.

Conference Day
Wed 26 May

Displayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change

11:20 - 12:20
2.1.2. Deep Neural Networks: Quality AssuranceTechnical Track at Blended Sessions Room 2 +12h
Chair(s): Gregorio RoblesUniversidad Rey Juan Carlos
11:20
20m
Paper
Graph-based Fuzz Testing for Deep Learning Inference EnginesTechnical Track
Technical Track
Weisi LuoI&V Dept of Kirin Solution Dept, HS, Huawei, Xiaoyue RunI&V Dept of Kirin Solution Dept, HS, Huawei, Dong ChaiI&V Dept of Kirin Solution Dept, HS, Huawei, Jiang WangI&V Dept of Kirin Solution Dept, HS, Huawei, Chunrong FangNanjing University, Zhenyu ChenNanjing University
Pre-print Media Attached File Attached
11:40
20m
Paper
RobOT: Robustness-Oriented Testing for Deep Learning SystemsTechnical Track
Technical Track
Jingyi WangZhejiang University, Jialuo ChenZhejiang University, Youcheng SunQueen's University Belfast, UK, Xingjun MaDeakin University, Dongxia WangZhejiang University, Jun SunSingapore Management University, Singapore, Peng ChengZhejiang University
Pre-print Media Attached
12:00
20m
Paper
Scalable Quantitative Verification For Deep Neural NetworksArtifact ReusableTechnical Track
Technical Track
Teodora BalutaNational University of Singapore, Zheng Leong ChuaIndependent Researcher, Kuldeep S. MeelNational University of Singapore, Prateek SaxenaNational University of Singapore
Pre-print Media Attached
23:20 - 00:20
2.1.2. Deep Neural Networks: Quality AssuranceTechnical Track at Blended Sessions Room 2
23:20
20m
Paper
Graph-based Fuzz Testing for Deep Learning Inference EnginesTechnical Track
Technical Track
Weisi LuoI&V Dept of Kirin Solution Dept, HS, Huawei, Xiaoyue RunI&V Dept of Kirin Solution Dept, HS, Huawei, Dong ChaiI&V Dept of Kirin Solution Dept, HS, Huawei, Jiang WangI&V Dept of Kirin Solution Dept, HS, Huawei, Chunrong FangNanjing University, Zhenyu ChenNanjing University
Pre-print Media Attached File Attached
23:40
20m
Paper
RobOT: Robustness-Oriented Testing for Deep Learning SystemsTechnical Track
Technical Track
Jingyi WangZhejiang University, Jialuo ChenZhejiang University, Youcheng SunQueen's University Belfast, UK, Xingjun MaDeakin University, Dongxia WangZhejiang University, Jun SunSingapore Management University, Singapore, Peng ChengZhejiang University
Pre-print Media Attached
00:00
20m
Paper
Scalable Quantitative Verification For Deep Neural NetworksArtifact ReusableTechnical Track
Technical Track
Teodora BalutaNational University of Singapore, Zheng Leong ChuaIndependent Researcher, Kuldeep S. MeelNational University of Singapore, Prateek SaxenaNational University of Singapore
Pre-print Media Attached