Write a Blog >>
ICSE 2021
Mon 17 May - Sat 5 June 2021

Despite the functional success of deep neural networks, their trustworthiness remains a crucial open challenge. To address this challenge, both testing and verification techniques have been proposed. But these existing techniques provide either scalability to large networks or formal guarantees, not both. In this paper, we propose a scalable quantitative verification framework for deep neural networks, i.e., a test-driven approach that comes with formal guarantees that a desired probabilistic property is satisfied. Our technique performs enough tests until soundness of a formal probabilistic property can be proven. It can be used to certify properties of both deterministic and randomized DNNs. We implement our approach in a tool called PROVERO and apply it in the context of certifying adversarial robustness of DNNs. In this context, we first show a new attack- agnostic measure of robustness which offers an alternative to purely attack-based methodology of evaluating robustness being reported today. Second, PROVERO provides certificates of robustness for large DNNs, where existing state-of-the-art verification tools fail to produce conclusive results. Our work paves the way forward for verifying properties of distributions captured by real-world deep neural network, with provable guarantees, even where testers only have black-box access to the neural network.

Conference Day
Wed 26 May

Displayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change

11:20 - 12:20
2.1.2. Deep Neural Networks: Quality AssuranceTechnical Track at Blended Sessions Room 2 +12h
Chair(s): Gregorio RoblesUniversidad Rey Juan Carlos
11:20
20m
Paper
Graph-based Fuzz Testing for Deep Learning Inference EnginesTechnical Track
Technical Track
Weisi LuoI&V Dept of Kirin Solution Dept, HS, Huawei, Xiaoyue RunI&V Dept of Kirin Solution Dept, HS, Huawei, Dong ChaiI&V Dept of Kirin Solution Dept, HS, Huawei, Jiang WangI&V Dept of Kirin Solution Dept, HS, Huawei, Chunrong FangNanjing University, Zhenyu ChenNanjing University
Pre-print Media Attached File Attached
11:40
20m
Paper
RobOT: Robustness-Oriented Testing for Deep Learning SystemsTechnical Track
Technical Track
Jingyi WangZhejiang University, Jialuo ChenZhejiang University, Youcheng SunQueen's University Belfast, UK, Xingjun MaDeakin University, Dongxia WangZhejiang University, Jun SunSingapore Management University, Singapore, Peng ChengZhejiang University
Pre-print Media Attached
12:00
20m
Paper
Scalable Quantitative Verification For Deep Neural NetworksArtifact ReusableTechnical Track
Technical Track
Teodora BalutaNational University of Singapore, Zheng Leong ChuaIndependent Researcher, Kuldeep S. MeelNational University of Singapore, Prateek SaxenaNational University of Singapore
Pre-print Media Attached
23:20 - 00:20
2.1.2. Deep Neural Networks: Quality AssuranceTechnical Track at Blended Sessions Room 2
23:20
20m
Paper
Graph-based Fuzz Testing for Deep Learning Inference EnginesTechnical Track
Technical Track
Weisi LuoI&V Dept of Kirin Solution Dept, HS, Huawei, Xiaoyue RunI&V Dept of Kirin Solution Dept, HS, Huawei, Dong ChaiI&V Dept of Kirin Solution Dept, HS, Huawei, Jiang WangI&V Dept of Kirin Solution Dept, HS, Huawei, Chunrong FangNanjing University, Zhenyu ChenNanjing University
Pre-print Media Attached File Attached
23:40
20m
Paper
RobOT: Robustness-Oriented Testing for Deep Learning SystemsTechnical Track
Technical Track
Jingyi WangZhejiang University, Jialuo ChenZhejiang University, Youcheng SunQueen's University Belfast, UK, Xingjun MaDeakin University, Dongxia WangZhejiang University, Jun SunSingapore Management University, Singapore, Peng ChengZhejiang University
Pre-print Media Attached
00:00
20m
Paper
Scalable Quantitative Verification For Deep Neural NetworksArtifact ReusableTechnical Track
Technical Track
Teodora BalutaNational University of Singapore, Zheng Leong ChuaIndependent Researcher, Kuldeep S. MeelNational University of Singapore, Prateek SaxenaNational University of Singapore
Pre-print Media Attached