An industrial experience report on applying search-based boundary input generation to cyber-physical systems
This program is tentative and subject to change.
Testing Cyber-Physical Systems (CPS) is crucial, as they play a central role in modern society and are deployed in several domains such as autonomous driving, robotics, and smart infrastructures. These systems integrate software controllers with physical processes, leading to highly complex input spaces and nonlinear behaviors. As a result, even small variations in operational conditions can cause large deviations in system performance, often degrading Quality of Service (QoS) or even leading to safety violations. Identifying such \textit{boundary scenarios}, inputs where minimal perturbations lead to large behavioral changes, is fundamental to ensure high system dependability prior to deploying a software version in operation.
However, generating boundary scenarios for CPSs poses several challenges: (i) the simulation-based execution of CPS test cases is computationally expensive; (ii) the input space is multi-dimensional and includes time-dependent signals; (iii) QoS requirements are difficult to formalize due to the complex outputs; and (iv) many automatically generated test inputs may be physically invalid. While prior work has addressed these challenges individually, no comprehensive solution exists that combines the test input minimization, behavioral boundary detection, and test input validity.
In this experience paper, we propose LiftJanus, the first search-based test generator for CPSs that integrates three state-of-the-art techniques into a unified framework. First, a test input minimization component that by means of delta-debugging reduces the test inputs while preserving a set of pre-defined properties. Second, a boundary detection algorithm, inspired by DeepJanus, generates input pairs that differ minimally but induce maximally different QoS outcomes, thereby exposing sensitive operating boundaries. Third, an automated repair component adjusts CPS configurations, ensuring that the produced test cases are physically feasible.
We empirically evaluated LiftJanus in two real-world elevator systems provided by our industrial partner, \Orona, one of the largest elevator manufacturers worldwide. The evaluation combined quantitative and qualitative analyses. Quantitative analysis showed that LiftJanus generated boundary inputs that were on average twice as effective as baseline techniques in degrading system QoS. The repair mechanism successfully improved configuration parameters in 76.25% of boundary cases. With respect to the qualitative assessment, open-ended interviews with domain experts confirmed that LiftJanus not only identifies scenarios previously overlooked by manual testing but also provides actionable insights for improving system configurations and dispatching algorithms. Experts stated that generating minimal yet impactful test cases is particularly valuable, as it reduces analysis overhead. Moreover, we grasped a set of lessons that can be beneficial to other practitioners in different domains.
In summary, this paper makes three key contributions: (1) We introduce LiftJanus, a novel comprehensive CPS testing framework. (2) We demonstrate, through an industrial case study, that LiftJanus scales to complex, real-world systems and produces significantly more effective and valid boundary test inputs than baselines. (3) We report practical evidence from domain experts, highlighting both the benefits of boundary-based testing and the challenges of transferring research prototypes into industrial practice.
This program is tentative and subject to change.
Wed 15 AprDisplayed time zone: Brasilia, Distrito Federal, Brazil change
11:00 - 12:30 | Testing and Analysis 2Journal-first Papers / SE In Practice (SEIP) / Research Track at Oceania II Chair(s): Andrea Stocco Technical University of Munich, fortiss | ||
11:00 15mTalk | An industrial experience report on applying search-based boundary input generation to cyber-physical systems Journal-first Papers Pablo Valle Mondragon University, Vincenzo Riccio University of Udine, Aitor Arrieta Mondragon University, Paolo Tonella USI Lugano, Maite Arratibel Orona | ||
11:15 15mTalk | Testing CPS with Design Assumptions-Based Metamorphic Relations and Genetic Programming Journal-first Papers Claudio Mandrioli University of Luxembourg, Seung Yeob Shin University of Luxembourg, Domenico Bianculli University of Luxembourg, Lionel Briand University of Ottawa, Canada; Lero centre, University of Limerick, Ireland Link to publication DOI Pre-print | ||
11:30 15mTalk | Signal Feature Coverage and Testing for CPS Dataflow Models Journal-first Papers Ezio Bartocci TU Wien, Leonardo Mariani University of Milano-Bicocca, Dejan Nickovic Austrian Institute of Technology, Drishti Yadav University of Luxembourg, Luxembourg | ||
11:45 15mTalk | Uncovering Failures in Cyber-Physical System State Transitions: A Fuzzing-Based Approach Applied to sUAS Research Track Theodore Chambers University of Notre Dame, Arturo Miguel Russell Bernal University of Notre Dame, Michael Vierhauser University of Innsbruck, Jane Cleland-Huang University of Notre Dame Pre-print | ||
12:00 15mTalk | Vision Language Model-based Testing of Industrial Autonomous Mobile Robots SE In Practice (SEIP) Jiahui Wu Simula Research Laboratory and University of Oslo, Chengjie Lu Simula Research Laboratory and University of Oslo, Aitor Arrieta Mondragon University, Shaukat Ali Simula Research Laboratory and Oslo Metropolitan University, Thomas Peyrucain PAL Robotics | ||
12:15 15mTalk | Misbehavior Forecasting for Focused Autonomous Driving Systems Testing Research Track M M Abid Naziri North Carolina State University, Stefano Carlo Lambertenghi Technische Universität München, fortiss GmbH, Andrea Stocco Technical University of Munich, fortiss, Marcelo d'Amorim North Carolina State University Pre-print | ||