Visual Language Action (VLA) models are multimodal robotic task controllers that, given an instruction and visual inputs, produce a sequence of low-level control actions (or motor commands) enabling a robot to execute the requested task in the physical environment. These systems face the test oracle problem from multiple perspectives. On the one hand, a test oracle must be defined for each instruction prompt, which is a complex and non-generalizable approach. On the other hand, current state-of-the-art oracles typically capture symbolic representations of the world (e.g., robot and object states), enabling the correctness evaluation of a task, but fail to assess other critical aspects, such as the quality with which VLA-enabled robots perform a task. In this paper, we explore whether Metamorphic Testing (MT) can alleviate the test oracle problem in this context. To do so, we propose two metamorphic relation patterns and five metamorphic relations to assess whether changes to the test inputs impact the original trajectory of the VLA-enabled robots. An empirical study involving five VLA models, two simulated robots, and four robotic tasks shows that MT can effectively alleviate the test oracle problem by automatically detecting diverse types of failures, including, but not limited to, uncompleted tasks. More importantly, the proposed MRs are generalizable, making the proposed approach applicable across different VLA models, robots, and tasks, even in the absence of test oracles.
Tue 19 MayDisplayed time zone: Seoul change
11:00 - 12:30 | Autonomous Systems & Robotics TestingIndustry / Research Papers at Room 101 Chair(s): Khouloud Gaaloul University of Michigan - Dearborn | ||
11:00 25mTalk | Dynasto: Validity-Aware Dynamic–Static Parameter Optimization for Autonomous Driving TestingDistinguished Paper Award Research Papers Dmytro Humeniuk Polytechnique Montréal, Mohammad Hamdaqa Polytechnique Montreal, Houssem Ben Braiek Polytechnique Montreal, Amel Bennaceur The Open University, UK, Foutse Khomh Polytechnique Montréal | ||
11:25 25mTalk | Natural Adversaries: Fuzzing Autonomous Vehicles with Realistic Roadside Object Placements Research Papers Yang Sun Singapore Management University, Haoyu Wang School of Computing and Information Systems, Singapore Management University, Chris Poskitt Singapore Management University, Jun Sun Singapore Management University DOI Pre-print | ||
11:50 15mTalk | Metamorphic Testing of Vision-Language Action–Enabled Robots Research Papers Pablo Valle Mondragon University, Sergio Segura SCORE Lab, I3US Institute, Universidad de Sevilla, Seville, Spain, Shaukat Ali Simula Research Laboratory and Oslo Metropolitan University, Aitor Arrieta Mondragon University Pre-print | ||
12:05 25mTalk | Assessing Vision–Language Models for Perception in Autonomous Underwater Robotic Software Industry Muhammad Yousaf Simula Research Laboratory, Aitor Arrieta Mondragon University, Shaukat Ali Simula Research Laboratory and Oslo Metropolitan University, Paolo Arcaini National Institute of Informatics, Shuai Wang DNV AS | ||