Generative AI in Simulation-Based Test Environments for Large-Scale Cyber-Physical Systems: An Industrial Study
Quality assurance for large-scale cyber-physical systems relies on sophisticated test activities using complex test environments investigated with the help of numerous types of simulators. As these systems grow, extensive resources are required to develop and maintain simulation models of hardware and software components, as well as physical environments. Meanwhile, recent advances in generative AI have led to tools that can produce executable test cases for software systems, offering potential benefits such as reducing manual efforts or increasing test coverage. However, the application of generative AI techniques to simulation-based testing of large-scale cyber-physical systems remains underexplored. To better understand this gap, this study captures practitioners’ perspectives on leveraging generative AI, based on a cross-company workshop with six organizations. Our contribution is twofold: (1) detailed, experience-based insights into challenges faced by engineers, and (2) a research agenda comprising three high-priority directions: (a) AI-generated scenarios and environment models, (b) simulators and AI in CI/CD pipelines, and (c) trustworthiness in generative AI for simulation. While participants acknowledged substantial potential, they also highlighted unresolved challenges. By detailing these issues, the paper aims to guide future academia-industry collaboration towards the responsible adoption of generative AI in simulation-based testing.
| Slides (PROFES2025_Presentation.pdf) | 153KiB |
Wed 3 DecDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
09:00 - 11:00 | Testing and Simulation of Software and Cyber-Physical SystemsShort Papers and Posters / Research Papers / Industry Papers at Sala degli Affreschi (Fresco Room) Chair(s): Javier Gonzalez-Huerta Blekinge Institute of Technology | ||
09:00 15mTalk | Coverage Isn’t Enough: SBFL-Driven Insights into Manually Created vs. Automatically Generated Tests Research Papers File Attached | ||
09:15 15mTalk | A Robust LSTM-based Test Selection Method for Self-Driving Cars Research Papers Ali Gullu University of Tartu, Faiz Ali Shah University of Tartu, Estonia, Dietmar Pfahl University of Tartu | ||
09:30 15mTalk | Generative AI in Simulation-Based Test Environments for Large-Scale Cyber-Physical Systems: An Industrial Study Research Papers Masoud Sadrnezhaad Linköping University, José Antonio Hernández López Universidad de Murcia, Torvald Mårtensson Saab AB, Daniel Varro Linköping University / McGill University DOI Pre-print File Attached | ||
09:45 10mTalk | Obtaining Test Data in the Estonian e-Government System: Challenges and Improvement Potential Industry Papers Maj-Annika Tammisto University of Tartu, Rudolf Ramler Software Competence Center Hagenberg (SCCH), Faiz Ali Shah University of Tartu, Estonia, Dietmar Pfahl University of Tartu | ||
09:55 10mTalk | Software Testing Education and Industry Needs - Report from ENACTEST EU Project Industry Papers Mehrdad Saadatmand RISE Research Institutes of Sweden, Muhammad Abbas Khan RISE Research Institutes of Sweden, Beatriz Marín Universitat Politècnica de València, Ana Paiva INESC TEC, Faculty of Engineering, University of Porto, Nele Van Asch AE nv, Graham Moran NEXO QA, Felix Cammaerts KU Leuven, Monique Snoeck Katholieke Universiteit Leuven, Alexandra Mendes Faculty of Engineering, University of Porto & INESC TEC Pre-print | ||
10:05 10mTalk | STRIPID: Simulation Test Ranking and Interactive Performance Inspection for PID controllers Industry Papers Alejandra Duque-Torres Software Competence Center Hagenberg (SCCH) GmbH, Claus Klammer Software Competence Center Hagenberg, Stefan Fischer Software Competence Center Hagenberg File Attached | ||
10:15 7mTalk | Lab Package Development as a Means for Educating Software Engineering Students Short Papers and Posters Eliisabet Kaasik University of Tartu, Faiz Ali Shah University of Tartu, Estonia, Dietmar Pfahl University of Tartu File Attached | ||
10:22 7mTalk | A Model-Driven Engineering Method for the Development of Digital Twins Short Papers and Posters Emilio Carrión Mercadona Tech, Mercadona - PROS – VRAIN Universitat Politècnica de València, Pedro Valderas Universitat Politècnica de València, Oscar Pastor Universitat Politecnica de Valencia | ||
10:29 7mTalk | From Scenario Selection to Simulation: Safety Testing of an Automated Driving System Short Papers and Posters Fauzia Khan University of Tartu, Estonia, Ali Gullu University of Tartu, Hina Anwar University of Tartu, Dietmar Pfahl University of Tartu | ||
10:36 7mTalk | An Application of Program Mutations For Generating Negative Test Scripts Mimicking Human Errors on Web Applications Short Papers and Posters Tomoya Yamashita Ehime University, Hirohisa Aman Ehime University, Sousuke Amasaki Nanzan University, Tomoyuki Yokogawa Okayama Prefectural University, Minoru Kawahara Ehime University | ||
10:43 7mTalk | Ticket-Augmented Just-in-Time Defect Prediction Short Papers and Posters Emanuele Gentili Mbda Italy S.p.a., Daniele La Prova University of Rome "Tor Vergata", Via del Politecnico 1, Rome, Italy, 00132, Falessi Davide University of Rome Tor Vergata | ||