ICST 2026
Mon 18 - Fri 22 May 2026 Daejeon, South Korea
Tue 19 May 2026 11:50 - 12:05 at Room 101 - Autonomous Systems & Robotics Testing Chair(s): Khouloud Gaaloul

Visual Language Action (VLA) models are multimodal robotic task controllers that, given an instruction and visual inputs, produce a sequence of low-level control actions (or motor commands) enabling a robot to execute the requested task in the physical environment. These systems face the test oracle problem from multiple perspectives. On the one hand, a test oracle must be defined for each instruction prompt, which is a complex and non-generalizable approach. On the other hand, current state-of-the-art oracles typically capture symbolic representations of the world (e.g., robot and object states), enabling the correctness evaluation of a task, but fail to assess other critical aspects, such as the quality with which VLA-enabled robots perform a task. In this paper, we explore whether Metamorphic Testing (MT) can alleviate the test oracle problem in this context. To do so, we propose two metamorphic relation patterns and five metamorphic relations to assess whether changes to the test inputs impact the original trajectory of the VLA-enabled robots. An empirical study involving five VLA models, two simulated robots, and four robotic tasks shows that MT can effectively alleviate the test oracle problem by automatically detecting diverse types of failures, including, but not limited to, uncompleted tasks. More importantly, the proposed MRs are generalizable, making the proposed approach applicable across different VLA models, robots, and tasks, even in the absence of test oracles.

Tue 19 May

Displayed time zone: Seoul change

11:00 - 12:30
Autonomous Systems & Robotics TestingIndustry / Research Papers at Room 101
Chair(s): Khouloud Gaaloul University of Michigan - Dearborn
11:00
25m
Talk
Dynasto: Validity-Aware Dynamic–Static Parameter Optimization for Autonomous Driving TestingDistinguished Paper AwardArtifact Available
Research Papers
Dmytro Humeniuk Polytechnique Montréal, Mohammad Hamdaqa Polytechnique Montreal, Houssem Ben Braiek Polytechnique Montreal, Amel Bennaceur The Open University, UK, Foutse Khomh Polytechnique Montréal
11:25
25m
Talk
Natural Adversaries: Fuzzing Autonomous Vehicles with Realistic Roadside Object Placements
Research Papers
Yang Sun Singapore Management University, Haoyu Wang School of Computing and Information Systems, Singapore Management University, Chris Poskitt Singapore Management University, Jun Sun Singapore Management University
DOI Pre-print
11:50
15m
Talk
Metamorphic Testing of Vision-Language Action–Enabled RobotsArtifact ReviewedArtifact Available
Research Papers
Pablo Valle Mondragon University, Sergio Segura SCORE Lab, I3US Institute, Universidad de Sevilla, Seville, Spain, Shaukat Ali Simula Research Laboratory and Oslo Metropolitan University, Aitor Arrieta Mondragon University
Pre-print
12:05
25m
Talk
Assessing Vision–Language Models for Perception in Autonomous Underwater Robotic Software
Industry
Muhammad Yousaf Simula Research Laboratory, Aitor Arrieta Mondragon University, Shaukat Ali Simula Research Laboratory and Oslo Metropolitan University, Paolo Arcaini National Institute of Informatics, Shuai Wang DNV AS