Write a Blog >>
Wed 12 Oct 2022 11:10 - 11:30 at Banquet A - Technical Session 10 - Testing I Chair(s): Gordon Fraser

Virtual personal assistant (VPA) services, e.g. Amazon Alexa and Google Assistant, are becoming increasingly popular recently. Users interact with them through voice-based apps, e.g. Amazon Alexa skills and Google Assistant actions. Unlike the desktop and mobile apps which have visible and intuitive graphical user interface (GUI) to facilitate interaction, VPA apps convey information purely verbally through the voice user interface (VUI), which is known to be limited in its invisibility, single mode and high demand of user attention. This may lead to various problems on the usability and correctness of VPA apps.

In this work, we propose a model-based framework named Vitas to handle VUI testing of VPA apps. Vitas interacts with the app VUI, and during the testing process, it retrieves semantic information from voice feedbacks by natural language processing. It incrementally constructs the finite state machine (FSM) model of the app with a weighted exploration strategy guided by key factors such as the coverage of app functionality. We conduct a large-scale testing on 41,581 VPA apps (i.e., skills) of Amazon Alexa, the most popular VPA service, and find that 51.29% of them have weaknesses. They largely suffer from problems such as unexpected exit/start, privacy violation and so on. Our work reveals the immaturity of the VUI designs and implementations in VPA apps, and sheds light on the improvement of several crucial aspects of VPA apps.

Wed 12 Oct

Displayed time zone: Eastern Time (US & Canada) change

10:00 - 12:00
Technical Session 10 - Testing IResearch Papers / Industry Showcase / Tool Demonstrations at Banquet A
Chair(s): Gordon Fraser University of Passau
Research paper
Inline Tests
Research Papers
Yu Liu University of Texas at Austin, Pengyu Nie University of Texas at Austin, Owolabi Legunsen Cornell University, Milos Gligoric University of Texas at Austin
LiveRef: a Tool for Live Refactoring Java Code
Tool Demonstrations
Sara Fernandes FEUP, Universidade do Porto, Ademar Aguiar FEUP, Universidade do Porto, André Restivo LIACC, Universidade do Porto, Porto, Portugal
Research paper
Sorry, I don't Understand: Improving Voice User Interface Testing
Research Papers
Emanuela Guglielmi University of Molise, Giovanni Rosa University of Molise, Simone Scalabrino University of Molise, Gabriele Bavota Software Institute, USI Università della Svizzera italiana, Rocco Oliveto University of Molise
Industry talk
MOREST: Industry Practice of Automatic RESTful API Testing
Industry Showcase
Yi Liu Nanyang Technological University, Yuekang Li Nanyang Technological University, Yang Liu Nanyang Technological University, Ruiyuan Wan , Runchao Wu Huawei Inc., Qingkun Liu Huawei Cloud Computing Technologies Co., Ltd
Research paper
VITAS : Guided Model-based VUI Testing of VPA AppsVirtual
Research Papers
Suwan Li Nanjing University, Lei Bu Nanjing University, Guangdong Bai University of Queensland, Zhixiu Guo Institute of Information Engineering, Chinese Academy of Sciences, China, Kai Chen SKLOIS, Institute of Information Engineering, Chinese Academy of Sciences, China, Hanlin Wei The University of Queensland