Write a Blog >>
ISSTA 2021
Sun 11 - Sat 17 July 2021 Online
co-located with ECOOP and ISSTA 2021

Automatically detecting the positions of key-points (e.g., facial key-points or finger key-points) in an image is an essential problem in many applications, such as driver's gaze detection and drowsiness detection in automated driving systems. With the recent advances of Deep Neural Networks (DNNs), Key-Points detection DNNs (KP-DNNs) have been increasingly employed for that purpose. Nevertheless, KP-DNN testing and validation have remained a challenging problem because KP-DNNs predict many independent key-points at the same time—where each individual key-point may be critical in the targeted application—and images can vary a great deal according to many factors.

In this paper, we present an approach to automatically generate test data for KP-DNNs using many-objective search. In our experiments, focused on facial key-points detection DNNs developed for an industrial automotive application, we show that our approach can generate test suites to severely mispredict, on average, more than 93% of all key-points. In comparison, random search-based test data generation can only severely mispredict 41% of them. Many of these mispredictions, however, are not avoidable and should not therefore be considered failures. We also empirically compare state-of-the-art, many-objective search algorithms and their variants, tailored for test suite generation. Furthermore, we investigate and demonstrate how to learn specific conditions, based on image characteristics (e.g., head posture and skin color), that lead to severe mispredictions. Such conditions serve as a basis for risk analysis or DNN retraining.

Conference Day
Wed 14 Jul

Displayed time zone: Brussels, Copenhagen, Madrid, Paris change

19:40 - 20:20
Session 4 (time band 1) Testing Deep Learning Systems 2Technical Papers at ISSTA 2
Chair(s): Sebastian ElbaumUniversity of Virginia
19:40
20m
Talk
DeepHyperion: Exploring the Feature Space of Deep Learning-Based Systems through Illumination Search
Technical Papers
Tahereh ZohdinasabUSI Lugano, Vincenzo RiccioUSI Lugano, Alessio GambiUniversity of Passau, Paolo TonellaUSI Lugano
DOI File Attached
20:00
20m
Talk
Automatic Test Suite Generation for Key-Points Detection DNNs using Many-Objective Search (Experience Paper)
Technical Papers
Fitash Ul HaqUniversity of Luxembourg, Donghwan ShinUniversity of Luxembourg, Lionel BriandUniversity of Luxembourg; University of Ottawa, Thomas StifterIEE, Jun WangPost Luxembourg
DOI

Conference Day
Thu 15 Jul

Displayed time zone: Brussels, Copenhagen, Madrid, Paris change

09:10 - 10:50
Session 9 (time band 3) Testing Deep Learning Systems 3Technical Papers at ISSTA 1
Chair(s): Mauro PezzeUSI Lugano; Schaffhausen Institute of Technology
09:10
20m
Talk
Attack as Defense: Characterizing Adversarial Examples using Robustness
Technical Papers
Zhe ZhaoShanghaiTech University, Guangke ChenShanghaiTech University, Jingyi WangZhejiang University, Yiwei YangShanghaiTech University, Fu SongShanghaiTech University, Jun SunSingapore Management University
DOI
09:30
20m
Talk
Exposing Previously Undetectable Faults in Deep Neural Networks
Technical Papers
Isaac DunnUniversity of Oxford, Hadrien PougetUniversity of Oxford, Daniel KroeningAmazon, Tom MelhamUniversity of Oxford
DOI Pre-print Media Attached
09:50
20m
Talk
Automatic Test Suite Generation for Key-Points Detection DNNs using Many-Objective Search (Experience Paper)
Technical Papers
Fitash Ul HaqUniversity of Luxembourg, Donghwan ShinUniversity of Luxembourg, Lionel BriandUniversity of Luxembourg; University of Ottawa, Thomas StifterIEE, Jun WangPost Luxembourg
DOI
10:10
20m
Talk
DeepHyperion: Exploring the Feature Space of Deep Learning-Based Systems through Illumination Search
Technical Papers
Tahereh ZohdinasabUSI Lugano, Vincenzo RiccioUSI Lugano, Alessio GambiUniversity of Passau, Paolo TonellaUSI Lugano
DOI File Attached
10:30
20m
Talk
DeepCrime: Mutation Testing of Deep Learning Systems Based on Real Faults
Technical Papers
Nargiz HumbatovaUSI Lugano, Gunel JahangirovaUSI Lugano, Paolo TonellaUSI Lugano
DOI