DeepTest 2021
Tue 1 Jun 2021
co-located with ICSE 2021
Tue 1 Jun 2021 11:00 - 11:20 at DeepTest Room - Session 1 Chair(s): Gunel Jahangirova, Andrea Stocco

Surprise Adequacy (SA) is one of the emerging and most promising adequacy criteria for Deep Learning (DL) testing. As an adequacy criterion, it has been used to assess the strength of DL test suites. In addition, it has also been used to find inputs to a Deep Neural Network (DNN) which were not sufficiently represented in the training data, or to select samples for DNN retraining. However, computation of the SA metric for a test suite can be prohibitively expensive, as it involves a quadratic number of distance calculations. Hence, we developed and released a performance-optimized, but functionally equivalent, implementation of SA, reducing the evaluation time by up to 97%. We also propose refined variants of the SA computation algorithm, aiming to further increase the evaluation speed. We then performed an empirical study on MNIST, focused on the out-of-distribution detection capabilities of SA, which allowed us to reproduce parts of the results presented when SA was first released. The experiments show that our refined variants are substantially faster than plain SA, while producing comparable outcomes. Our experimental results exposed also an overlooked issue of SA: it can be highly sensitive to the non-determinism associated with the DNN training procedure.

Tue 1 Jun

Displayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change

10:00 - 12:00
Session 1deeptest2021 at DeepTest Room
Chair(s): Gunel Jahangirova USI Lugano, Switzerland, Andrea Stocco Università della Svizzera italiana (USI)
Problem Solving Combining Data Science and Web Knowledge
Amir Ronen SparkBeyond
A Review and Refinement of Surprise Adequacy
Michael Weiss Università della Svizzera Italiana (USI), Rwiddhi Chakraborty USI Lugano, Switzerland, Paolo Tonella USI Lugano, Switzerland
Deep Learning-Based Prediction of Test Input Validity for RESTful APIs
Agatino Giuliano Mirabella Universidad de Sevilla, Alberto Martin-Lopez Universidad de Sevilla, Sergio Segura Universidad de Sevilla, Luis Valencia-Cabrera Universidad de Sevilla, Antonio Ruiz-Cortés University of Seville
Live Q&A
Open Discussion & Q/A

Information for Participants
Tue 1 Jun 2021 10:00 - 12:00 at DeepTest Room - Session 1 Chair(s): Gunel Jahangirova, Andrea Stocco
Info for room DeepTest Room:

Go directly to this room on Clowdr