ICST 2023
Sun 16 - Thu 20 April 2023 Dublin, Ireland
Mon 17 Apr 2023 17:00 - 17:20 at Grand canal - Session 5: Testing AI/ML systems Chair(s): Jie M. Zhang

Testing Deep Learning (DL) systems is a complex task as they do not behave like traditional systems would, notably because of their stochastic nature. Nonetheless, being able to adapt existing testing techniques such as Mutation Testing (MT) to DL settings would greatly improve their potential verifiability. While some efforts have been made to extend MT to the Supervised Learning paradigm (SL), little work has gone into extending it to Reinforcement Learning (RL) which is also an important component of the DL ecosystem but behaves very differently from SL. This paper builds on the existing approach of MT in order to propose a framework, RLMutation, for MT applied to RL. Notably, we use existing taxonomies of faults to build a set of mutation operators relevant to RL and use a simple heuristic to generate test cases for RL. This allows us to compare different MT methods based on existing approaches, as well as to analyze the behavior of the obtained mutation operators and their potential combinations called Higher Order Mutation (HOM). We show that the design choice of the MT method can affect whether or not a mutation is killed as well as the generated test cases. Moreover, we found that even with a relatively small number of test cases and operators we manage to generate HOMs with interesting properties which can enhance testing capability in RL systems.

Mon 17 Apr

Displayed time zone: Dublin change

16:00 - 18:00
Session 5: Testing AI/ML systemsResearch Papers / Previous Editions at Grand canal
Chair(s): Jie M. Zhang King's College London
16:00
20m
Talk
Robustness assessment and improvement of a neural network for blood oxygen pressure estimation
Previous Editions
Paolo Arcaini National Institute of Informatics , Andrea Bombarda University of Bergamo, Silvia Bonfanti University of Bergamo, Angelo Gargantini University of Bergamo, Daniele Gamba AISent S.r.l., Rita Pedercini AISent S.r.l.
DOI
16:20
20m
Talk
An Empirical Evaluation of Mutation Operators for Deep Learning Systems
Previous Editions
Gunel Jahangirova King's College London, Paolo Tonella USI Lugano
DOI
16:40
20m
Talk
Distributed Repair of Deep Neural Networks
Research Papers
Davide Li Calsi Politecnico di Milano, Matias Duran National Institute of Informatics, Xiao-Yi Zhang School of Computer and Communication Engineering, University of Science and Technology Beijing, Paolo Arcaini National Institute of Informatics , Fuyuki Ishikawa National Institute of Informatics
17:00
20m
Talk
Mutation Testing of Deep Reinforcement Learning Based on Real Faults
Research Papers
Florian Tambon Polytechnique Montréal, Vahid Majdinasab Polytechnique Montréal, Amin Nikanjam École Polytechnique de Montréal, Foutse Khomh Polytechnique Montréal, Giuliano Antoniol Polytechnique Montréal
Pre-print
17:20
20m
Talk
Repairing DNN Architecture: Are We There Yet?
Research Papers
Jinhan Kim KAIST, Nargiz Humbatova USI Lugano, Gunel Jahangirova King's College London, Paolo Tonella USI Lugano, Shin Yoo KAIST
Pre-print