Write a Blog >>
ICSE 2020
Wed 24 June - Thu 16 July 2020
Sat 11 Jul 2020 00:36 - 00:48 at Goguryeo - P26-Deep Learning Testing and Debugging Chair(s): Tim Menzies

Image classifiers have become an important component of today’s software, from consumer and business applications to safety-critical domains. The advent of Deep Neural Networks (DNNs) is the key catalyst behind such wide-spread success. However, wide adoption comes with serious concerns about the robustness of software systems dependent on image classification DNNs, as several severe erroneous behaviors have been reported under sensitive and critical circumstances. We argue that developers need to rigorously test their software’s image classifiers and delay deployment until acceptable. We present an approach to testing image classifier robustness based on class property violations.

We have found that many of the reported erroneous cases in popular DNN image classifiers occur because the trained models confuse one class with another or show biases towards some classes over others. These bugs usually violate some class properties of one or more of those classes. Most DNN testing techniques focus on per-image violations and thus fail to detect such class-level confusions or biases.

We developed a testing approach to automatically detect class-based confusion and bias errors in DNN-driven image classification software. We evaluated our implementation, DeepInspect, on several popular image classifiers with precision up to 100% (avg. 72.6%) for confusion errors, and up to 84.3% (avg. 66.8%) for bias errors. DeepInspect found hundreds of classification mistakes in widely-used models, many of which expose errors indicating confusion or bias.

Sat 11 Jul

Displayed time zone: (UTC) Coordinated Universal Time change

00:00 - 01:00
P26-Deep Learning Testing and DebuggingTechnical Papers at Goguryeo
Chair(s): Tim Menzies North Carolina State University
00:00
12m
Talk
ReluDiff: Differential Verification of Deep Neural NetworksArtifact ReusableTechnical
Technical Papers
Brandon Paulsen University of Southern California, Jingbo Wang University of Southern California, Chao Wang USC
Pre-print
00:12
12m
Talk
Structure-Invariant Testing for Machine TranslationTechnical
Technical Papers
Pinjia He ETH Zurich, Clara Meister ETH Zurich, Zhendong Su ETH Zurich, Switzerland
00:24
12m
Talk
Automatic Testing and Improvement of Machine TranslationTechnical
Technical Papers
Zeyu Sun Peking University, Jie M. Zhang University College London, UK, Mark Harman Facebook and University College London, Mike Papadakis University of Luxembourg, Lu Zhang Peking University, China
00:36
12m
Talk
Testing DNN Image Classifier for Confusion & Bias ErrorsArtifact ReusableTechnicalArtifact Available
Technical Papers
Yuchi Tian Columbia University, Ziyuan Zhong Columbia University, Vicente Ordonez University of Virginia, Gail Kaiser Columbia University, Baishakhi Ray Columbia University, New York
00:48
12m
Talk
Repairing Deep Neural Networks: Fix Patterns and ChallengesArtifact ReusableTechnicalArtifact Available
Technical Papers
Md Johirul Islam Iowa State University, Rangeet Pan Iowa State University, USA, Giang Nguyen Dept. of Computer Science, Iowa State University, Hridesh Rajan Iowa State University, USA