Learning Non-robustness using Simulation-based Testing: a Network Traffic-shaping Case Study
An input to a system reveals a non-robust behaviour when, by making a small change in the input, the output of the system changes from acceptable (passing) to unacceptable (failing) or vice versa. Identifying inputs that lead to non-robust behaviours is important for many types of systems, e.g., cyber-physical and network systems, whose inputs are prone to perturbations. In this paper, we propose an approach that combines simulation-based testing with regression tree models to generate value ranges for inputs in response to which a system is likely to exhibit non-robust behaviours. We apply our approach to a network traffic-shaping system (NTSS) – a novel case study from the network domain. In this case study, developed and conducted in collaboration with a network solutions provider, RabbitRun Technologies, input ranges that lead to non-robustness are of interest as a way to identify and mitigate network quality-of-service issues. We demonstrate that our approach accurately characterizes non-robust test inputs of NTSS by achieving a precision of 84% and a recall of 100%, significantly outperforming a standard baseline. In addition, we show that there is no statistically significant difference between the results obtained from our simulated testbed and a hardware testbed with identical configurations. Finally we describe lessons learned from our industrial collaboration, offering insights about how simulation helps discover unknown and undocumented behaviours as well as a new perspective on using non-robustness as a measure for system re-configuration.
Mon 17 AprDisplayed time zone: Dublin change
16:00 - 18:00 | Session 7: Testing for Safery & Security Industry / Research Papers / Journal-First Papers / Previous Editions at Hanover Chair(s): Eric Bodden Heinz Nixdorf Institut, Paderborn University and Fraunhofer IEM | ||
16:00 20mTalk | Learning Non-robustness using Simulation-based Testing: a Network Traffic-shaping Case Study Industry Baharin Aliashrafi Jodat University of Ottawa, Shiva Nejati University of Ottawa, Mehrdad Sabetzadeh University of Ottawa, Patricio Saavedra RabbitRun Technologies Inc Pre-print | ||
16:20 20mTalk | Test environments for large-scale software systems – an industrial study of intrinsic and extrinsic success factors Journal-First Papers | ||
16:40 20mTalk | Assessing the Effectiveness of Input and Output Coverage Criteria for Testing Quantum Programs Previous Editions Shaukat Ali Simula Research Laboratory, Paolo Arcaini National Institute of Informatics
, Xinyi Wang , Tao Yue Simula Research Laboratory DOI | ||
17:00 20mTalk | Heap Fuzzing: Automatic Garbage Collection Testing with Directed Random Events Research Papers Guillermo Polito Inria, Cristal, UMR 9189, Université de Lille, Pablo Tesone Univ. Lille, Inria, CNRS, Centrale Lille, UMR 9189 CRIStAL, Pharo Consortium, Jean Privat Université du Québec à Montréal (UQAM), Nahuel Palumbo Université Lille, CNRS, Centrale Lille, Inria, UMR 9189 - CRIStAL, Stéphane Ducasse Inria; University of Lille; CNRS; Centrale Lille; CRIStAL | ||
17:20 20mTalk | MagicMirror: Towards High-Coverage Fuzzing of Smart Contracts Research Papers Huadong Feng University of Texas at Arlington, Xiaolei Ren University of Texas at Arlington, Qiping Wei University of Texas at Arlington, Jeff Yu Lei University of Texas at Arlington, Raghu Kacker National Institute of Standards and Technology, Richard Kuhn National Institute of Standards and Technology, Dimitris Simos SBA Research |