Improving Random GUI Testing with Image-based Widget Detection
Graphical User Interfaces (GUIs) are one of the most common user interfaces, enabling interactions with applications through mouse movements and key presses. Tools for automated testing of GUIs exist, however they usually rely on operating system specific or framework specific knowledge in order to interact with the application under test. Because of frequent operating system updates and a large variety of GUI frameworks, such tools are made obsolete by time. Applications that use GUIs then need to find alternative tools to generate new tests, which again need some form of guidance for simulated interaction with the application. For an automated GUI test generation tool, supporting many frameworks and operating systems is impractical; new operating system updates can remove required information and each different GUI framework uses unique underlying data structures. We propose a technique for improving random GUI testing by automatically identifying GUI widgets in screen shots using machine learning techniques. This information provides guidance to GUI testing tools in environments not currently supported by deriving GUI widget information from screen shots only. In our experiments, we found that identifying GUI widgets from screen shots and using this information to guide random testing achieved a significantly higher branch coverage in 18 of 20 applications, with an average increase of 42.5% compared to conventional random testing.
Fri 19 JulDisplayed time zone: Beijing, Chongqing, Hong Kong, Urumqi change
14:00 - 15:30 | |||
14:00 22mTalk | TestMig: Migrating GUI Test Cases from iOS to Android Technical Papers Xue Qin , Hao Zhong Shanghai Jiao Tong University, Xiaoyin Wang University of Texas at San Antonio, USA | ||
14:22 22mTalk | Learning User Interface Element Interactions Technical Papers Christian Degott CISPA Helmholtz Center for Information Security, Nataniel Borges Jr. CISPA Helmholtz Center for Information Security, Andreas Zeller CISPA Helmholtz Center for Information Security Pre-print Media Attached | ||
14:45 22mTalk | Improving Random GUI Testing with Image-based Widget Detection Technical Papers Thomas D. White The University of Sheffield, Gordon Fraser University of Passau, Guy J. Brown The University of Sheffield | ||
15:07 22mTalk | Automatically Testing Self-Driving Cars with Search-based Procedural Content Generation Technical Papers |