GUIPilot: A Consistency-based Mobile GUI Testing Approach for Detecting Application-specific Bugs
GUI testing is crucial for ensuring the reliability of mobile applications. State-of-the-art GUI testing approaches are successful in exploring more application scenarios and discovering \textit{general} bugs such as application crashes. However, industrial GUI testing also needs to investigate \textit{application-specific} bugs such as deviations in screen layout, widget position, or GUI transition from the GUI design mock-ups created by the application designers. These mock-ups specify the expected screens, widgets, and their respective behaviors. Validating the consistency between the GUI design and the implementation is labor-intensive and time-consuming, yet, this validation step plays an important role in industrial GUI testing.
In this work, we propose GUIPilot, an approach for detecting inconsistencies between the mobile design and their implementations. The mobile design usually consists of design mock-ups that specify (1) the expected screen appearances (e.g., widget layouts, colors, and shapes) and (2) the expected screen behaviors, regarding how one screen can transition into another (e.g., labeled widgets with textual description). Given a design mock-up and the implementation of its application, GUIPilot reports both their screen inconsistencies as well as process inconsistencies. On the one hand, GUIPilot detects the screen inconsistencies by abstracting every screen into a widget container where each widget is represented by its position, width, height, and type. By defining the partial order of widgets and the costs of replacing, inserting, and deleting widgets in a screen, we convert the screen-matching problem into an optimizable widget alignment problem. On the other hand, we translate the specified GUI transition into stepwise actions on the mobile screen (e.g., click, long-press, input text on some widgets). To this end, we propose a \textit{visual prompt} for the vision-language model to infer widget-specific actions on the screen. By this means, we can validate the presence or absence of expected transitions in the implementation. Our extensive experiments on 80 mobile applications and 160 design mock-ups show that (1) GUIPilot can achieve 94.5% precision and 99.6% recall in detecting screen inconsistencies, outperforming the state-of-the-art approach, such as GVT, by 66.2% and 56.6% respectively, and (2) GUIPilot reports zero errors in detecting process inconsistencies. Furthermore, our industrial case study on applying GUIPilot on a trading mobile application shows that GUIPilot has detected nine application bugs, and all the bugs were confirmed by the original application experts.
GUIPilot video (video.mp4) | 7.90MiB |
Thu 26 JunDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
14:00 - 15:15 | |||
14:00 25mTalk | Are Autonomous Web Agents good testers? Research Papers Antoine Chevrot Smartesting, Alexandre Vernotte Smartesting, Jean-Rémy Falleri Univ. Bordeaux, CNRS, Bordeaux INP, LaBRI, UMR 5800, Institut Universitaire de France, Xavier Blanc Université de Bordeaux, Bruno Legeard Université de Bourgogne Franche-Comté and Smartesting, Aymeric Cretin Smartesting DOI | ||
14:25 25mTalk | MLLM-Based UI2Code Automation Guided by UI Layout Information Research Papers Fan Wu Harbin Institute of Technology, Shenzhen, Cuiyun Gao Harbin Institute of Technology, Shuqing Li The Chinese University of Hong Kong, Xin-Cheng Wen Harbin Institute of Technology, Qing Liao Harbin Institute of Technology DOI Pre-print | ||
14:50 25mTalk | GUIPilot: A Consistency-based Mobile GUI Testing Approach for Detecting Application-specific Bugs Research Papers Ruofan Liu Shanghai Jiao Tong University; National University of Singapore, Xiwen Teoh National University of Singapore, Yun Lin Shanghai Jiao Tong University, Guanjie Chen Shanghai Jiao Tong University, Ruofei Ren Shanghai Jiao Tong University, Denys Poshyvanyk William & Mary, Jin Song Dong National University of Singapore Link to publication DOI File Attached |
Cosmos 3C is the third room in the Cosmos 3 wing.
When facing the main Cosmos Hall, access to the Cosmos 3 wing is on the left, close to the stairs. The area is accessed through a large door with the number “3”, which will stay open during the event.