GUISpector: An MLLM Agent Framework for Automated Verification of Natural Language Requirements in GUI Prototypes
Graphical user interfaces (GUIs) are foundational to interactive systems and play a pivotal role in early requirements elicitation through prototyping. Ensuring that GUI implementations fulfill natural language (NL) requirements is essential for robust software engineering, especially as LLM-driven programming agents become increasingly integrated into development workflows. Existing GUI testing approaches, whether traditional or LLM-driven, often fall short in handling the complexity of modern interfaces, and typically lack actionable feedback and effective integration with automated development agents. In this paper, we introduce GUISpector, a novel framework that leverages a multi-modal (M)LLM-based agent for the automated verification of NL requirements in GUI prototypes. First, GUISpector adapts a MLLM agent to interpret and operationalize NL requirements, enabling to autonomously plan and execute verification trajectories across GUI applications. Second, GUISpector systematically extracts detailed NL feedback from the agent’s verification process, providing developers with actionable insights that can be used to iteratively refine the GUI artifact or directly inform LLM-based code generation in a closed feedback loop. Third, we present an integrated tool that unifies these capabilities, offering practitioners an accessible interface for supervising verification runs, inspecting agent rationales and managing the end-to-end requirements verification process. We evaluated GUISpector on a comprehensive set of 150 requirements based on 900 acceptance criteria annotations across diverse GUI applications, demonstrating effective detection of requirement satisfaction and violations and highlighting its potential for seamless integration of actionable feedback into automated LLM-driven development workflows. The video presentation of GUISpector is available at: https://youtu.be/JByYF6BNQeE, showcasing its main capabilities.
Thu 16 AprDisplayed time zone: Brasilia, Distrito Federal, Brazil change
11:00 - 12:30 | Testing and Analysis 9Research Track / Journal-first Papers / Demonstrations / New Ideas and Emerging Results (NIER) at Oceania II Chair(s): Shiyi Wei University of Texas at Dallas | ||
11:00 15mTalk | GUISpector: An MLLM Agent Framework for Automated Verification of Natural Language Requirements in GUI Prototypes Demonstrations Kristian Kolthoff Institute for Software and Systems Engineering, Clausthal University of Technology, Felix Kretzer human-centered systems Lab (h-lab), Karlsruhe Institute of Technology (KIT) , Simone Paolo Ponzetto Data and Web Science Group, University of Mannheim, Alexander Maedche human-centered systems Lab (h-lab), Karlsruhe Institute of Technology (KIT) , Christian Bartelt Institute for Software and Systems Engineering, TU Clausthal Pre-print Media Attached | ||
11:15 15mTalk | Valg: A Fast Reinforcement Learning-Based Runtime Verification Tool for Java Demonstrations Shinhae Kim Cornell University, Saikat Dutta Cornell University, Owolabi Legunsen Cornell University | ||
11:30 15mTalk | Quantum Neural Network Classifier for Cancer Registry System Testing: A Feasibility Study Journal-first Papers Xinyi Wang Simula Research Laboratory; University of Oslo, Shaukat Ali Simula Research Laboratory and Oslo Metropolitan University, Paolo Arcaini National Institute of Informatics, Narasimha Raghavan Veeraragavan Cancer Registry of Norway and Norwegian Institute of Public Health, Jan F. Nygård Cancer Registry of Norway Link to publication DOI | ||
11:45 15mTalk | Testora: Using Natural Language Intent to Detect Behavioral Regressions Research Track Michael Pradel CISPA Helmholtz Center for Information Security | ||
12:00 15mTalk | Automatic Validation of LLM-Generated Code with Prompt Paraphrasing New Ideas and Emerging Results (NIER) | ||
12:15 15mTalk | Causally Perturbed Fairness Testing Journal-first Papers | ||