Improving Random GUI Testing with Image-based Widget Detection
Graphical User Interfaces (GUIs) are one of the most common user interfaces, enabling interactions with applications through mouse movements and key presses. Tools for automated testing of GUIs exist, however they usually rely on operating system specific or framework specific knowledge in order to interact with the application under test. Because of frequent operating system updates and a large variety of GUI frameworks, such tools are made obsolete by time. Applications that use GUIs then need to find alternative tools to generate new tests, which again need some form of guidance for simulated interaction with the application. For an automated GUI test generation tool, supporting many frameworks and operating systems is impractical; new operating system updates can remove required information and each different GUI framework uses unique underlying data structures. We propose a technique for improving random GUI testing by automatically identifying GUI widgets in screen shots using machine learning techniques. This information provides guidance to GUI testing tools in environments not currently supported by deriving GUI widget information from screen shots only. In our experiments, we found that identifying GUI widgets from screen shots and using this information to guide random testing achieved a significantly higher branch coverage in 18 of 20 applications, with an average increase of 42.5% compared to conventional random testing.
Fri 19 Jul
|14:00 - 14:22|
|14:22 - 14:45|
Christian DegottCISPA Helmholtz Center for Information Security, Nataniel Borges Jr.CISPA Helmholtz Center for Information Security, Andreas ZellerCISPA Helmholtz Center for Information SecurityPre-print Media Attached
|14:45 - 15:07|
|15:07 - 15:30|