TCSE logo 
 Sigsoft logo
Sustainability badge
Wed 30 Apr 2025 11:30 - 11:45 at 213 - Autonomy Chair(s): Lionel Briand

Thorough simulation testing is crucial for validating the correct behavior of small Uncrewed Aerial Systems (sUAS) across multiple scenarios, including adverse weather conditions (such as wind, and fog), diverse settings (hilly terrain, or urban areas), and varying mission profiles (surveillance, tracking). While various sUAS simulation tools exist to support developers, the entire process of creating, executing, and analyzing simulation tests remains a largely manual and cumbersome task. Developers must identify test scenarios, set up the simulation environment, integrate the System under Test (SuT) with simulation tools, formulate mission plans, and collect and analyze results. These labor-intensive tasks limit the ability of developers to conduct exhaustive testing across a wide range of scenarios. To alleviate this problem, in this paper, we propose AUTOSIMTEST, a Large Language Model (LLM)-driven framework, where multiple LLM agents collaborate to support the sUAS simulation testing process. This includes: (1) creating test scenarios that subject the SuT to unique environmental contexts; (2) preparing the simulation environment as per the test scenario; (3) generating diverse sUAS missions for the SuT to execute; and (4) automatically analyzing simulation results and providing an interactive analytics interface. Further, the design of the framework is flexible for creating and testing scenarios for a variety of sUAS use cases, simulation tools, and SuT input requirements. We evaluated our approach by (a) conducting simulation testing of PX4 and ArduPilot flight-controller-based SuTs, (b) analyzing the performance of each agent, and (c) gathering feedback from sUAS developers. Our findings indicate that AUTOSIMTEST significantly improves the efficiency and scope of the sUAS testing process, allowing for more comprehensive and varied scenario evaluations while reducing the manual effort.

Wed 30 Apr

Displayed time zone: Eastern Time (US & Canada) change

11:00 - 12:30
AutonomyResearch Track at 213
Chair(s): Lionel Briand University of Ottawa, Canada; Lero centre, University of Limerick, Ireland
11:00
15m
Talk
A Differential Testing Framework to Identify Critical AV Failures Leveraging Arbitrary InputsArtifact-FunctionalArtifact-Available
Research Track
Trey Woodlief University of Virginia, Carl Hildebrandt University of Virginia, Sebastian Elbaum University of Virginia
11:15
15m
Talk
Automating a Complete Software Test Process Using LLMs: An Automotive Case Study
Research Track
Shuai Wang , Yinan Yu Chalmers University of Technology, Robert Feldt Chalmers | University of Gothenburg, Dhasarathy Parthasarathy Volvo Group
Pre-print
11:30
15m
Talk
LLM-Agents Driven Automated Simulation Testing and Analysis of small Uncrewed Aerial Systems
Research Track
Venkata Sai Aswath Duvvuru Saint Louis University, Bohan Zhang Saint Louis University, Missouri, Michael Vierhauser University of Innsbruck, Ankit Agrawal Saint Louis University, Missouri
Pre-print Media Attached
11:45
15m
Talk
Efficient Domain Augmentation for Autonomous Driving Testing Using Diffusion ModelsArtifact-FunctionalArtifact-AvailableArtifact-Reusable
Research Track
Luciano Baresi Politecnico di Milano, Davide Yi Xian Hu Politecnico di Milano, Andrea Stocco Technical University of Munich, fortiss, Paolo Tonella USI Lugano
Pre-print
12:00
15m
Talk
GARL: Genetic Algorithm-Augmented Reinforcement Learning to Detect Violations in Marker-Based Autonomous Landing Systems
Research Track
Linfeng Liang Macquarie University, Yao Deng Macquarie University, Kye Morton Skyy Network, Valtteri Kallinen Skyy Network, Alice James Macquarie University, Avishkar Seth Macquarie University, Endrowednes Kuantama Macquarie University, Subhas Mukhopadhyay Macquarie University, Richard Han Macquarie University, Xi Zheng Macquarie University
12:15
15m
Talk
Decictor: Towards Evaluating the Robustness of Decision-Making in Autonomous Driving SystemsArtifact-FunctionalArtifact-AvailableArtifact-Reusable
Research Track
Mingfei Cheng Singapore Management University, Xiaofei Xie Singapore Management University, Yuan Zhou Zhejiang Sci-Tech University, Junjie Wang Tianjin University, Guozhu Meng Institute of Information Engineering, Chinese Academy of Sciences, Kairui Yang DAMO Academy, Alibaba Group, China
:
:
:
: