From GUI Tests to Conversational Interaction: A New Perspective on App-Specific Voice Assistants
Voice assistants are widely deployed on mobile platforms, yet most are designed as system-level services that remain poorly aligned with application-specific behavior. As a result, enabling voice interaction at the app level requires developers to manually reimplement application logic and interaction workflows, leading to high development and maintenance costs.
We propose a novel, LLM-driven approach to automating app-specific voice assistants by repurposing GUI test code, which already encodes behavior-faithful, executable specifications of application functionality. In this paper, we advance a new perspective in which large language models act as semantic translators, reinterpreting GUI tests as bridges between application behavior and conversational interaction. By transforming test methods into app-specific VA artifacts, such as voice intents, action definitions, and executable interaction plans, our approach grounds voice assistants directly in existing application logic rather than external specifications.
We illustrate this vision through AppVA, a research prototype system that operationalizes the idea on Android. Our preliminary experience across five open-source applications suggests that GUI test code can be systematically reused beyond verification, enabling the synthesis of app-specific voice assistants and pointing toward a broader research direction at the intersection of software testing, interaction design, and LLM-enabled automation.
| Pre-print (AppVA_IVR_FSE_2026_camera_ready.pdf) | 1.57MiB |
Thu 9 JulDisplayed time zone: Eastern Time (US & Canada) change
14:00 - 15:20 | GUI TestingResearch Papers / Journal-First Paper / Ideas, Visions and Reflections at MB 2.430 Chair(s): Shin Yoo KAIST | ||
14:00 20mTalk | VLM-Fuzz: Vision Language Model Assisted Recursive Depth-First Search Exploration for Effective GUI Testing of Android Apps Journal-First Paper Biniam Fisseha Demissie Technology Innovation Institute, Abu Dhabi, UAE, Yan Naing Tun Singapore Management University, Lwin Khin Shar Singapore Management University, Mariano Ceccato University of Verona Link to publication DOI | ||
14:20 20mTalk | From Suspicious Signals to Crashes: Guiding Bug-driven GUI Testing via Code-inspired Tracing Research Papers Mengzhuo Chen Institute of Software, Chinese Academy of Sciences, Zhe Liu Institute of Software, Chinese Academy of Sciences, Chunyang Chen TU Munich, Junjie Wang Institute of Software at Chinese Academy of Sciences, Boyu Wu Institute of Software at Chinese Academy of Sciences, Yuekai Huang Institute of Software, Chinese Academy of Sciences, Jun Hu Institute of Software, Chinese Academy of Sciences, Qing Wang Institute of Software at Chinese Academy of Sciences | ||
14:40 20mTalk | WebTestPilot: Agentic End-to-End Web Testing against Natural Language Specification by Inferring Oracles with Symbolized GUI Elements Research Papers Xiwen Teoh National University of Singapore, Yun Lin Shanghai Jiao Tong University, Duc-Minh Nguyen Shanghai Jiao Tong University, Ruofei Ren Shanghai Jiao Tong University, Wenjie Zhang National University of Singapore, Jin Song Dong National University of Singapore Pre-print | ||
15:00 10mTalk | From GUI Tests to Conversational Interaction: A New Perspective on App-Specific Voice Assistants Ideas, Visions and Reflections Media Attached File Attached | ||
15:10 20mTalk | TUSR: A Test Unit–Based Framework for Repairing Obsolete GUI Test Scripts Research Papers | ||