Practical Non-Intrusive GUI Exploration Testing with Visual-based Robotic Arms
Graphical User Interface (GUI) testing has been a significant topic in the software engineering community. Most existing GUI testing frameworks are intrusive and can only support some specific platforms, which are quite limited. With the development of distinct scenarios, diverse embedded systems or customized operating systems on different devices do not support existing intrusive GUI testing frameworks. Some approaches adopt robotic arms to replace the interface invoking of mobile apps under test and use computer vision technologies to identify GUI elements. However, Some challenges remain unsolved with such approaches. First, existing approaches assume that GUI screens are fixed so that they cannot be adapted to diverse systems with different screen conditions. Second, existing approaches use XY-plane robotic arm system, which cannot flexibly simulate human testing operations. Third, existing approaches ignore the compatibility bugs of apps and only focus on the crash bugs. To sum up, a more practical approach is required for the non-intrusive scenario.
In order to solve the remaining challenges, we propose a practical non-intrusive GUI testing framework with visual-based robotic arms, namely RoboTest. RoboTest integrates a set of novel GUI screen and widget detection algorithm that is adaptive to detecting screens of different sizes and then to extracting GUI widgets from the detected screens. Then, a complete set of widely-used testing operations are applied with a 4-DOF robotic arm, which can more effectively and flexibly simulate human testing operations. During the app exploration, RoboTest integrates the specially designed Principle of Proximity-guided (PoP-guided) exploration strategy, which chooses close widgets of the previous operation targets to reduce the robotic arm movement overhead and improve exploration efficiency. Moreover, RoboTest can effectively detect some compatibility bugs beyond crash bugs with a GUI comparison on different devices of the same test operations. We evaluate RoboTest with 20 real-world mobile apps, together with a case study on a representative industrial embedded system. The results show that RoboTest can effectively, efficiently, and generally explore the AUT to find bugs and reduce app exploration time overhead from the robotic arm movement.
Thu 18 AprDisplayed time zone: Lisbon change
11:00 - 12:30 | Testing 3Research Track / Journal-first Papers / Software Engineering in Practice at Grande Auditório Chair(s): José Miguel Rojas The University of Sheffield | ||
11:00 15mTalk | Do Automatic Test Generation Tools Generate Flaky Tests? Research Track Martin Gruber BMW Group, University of Passau, Muhammad Firhard Roslan University of Sheffield, Owain Parry The University of Sheffield, Fabian Scharnböck University of Passau, Phil McMinn University of Sheffield, Gordon Fraser University of Passau Pre-print | ||
11:15 15mTalk | Deep Combination of CDCL(T) and Local Search for Satisfiability Modulo Non-Linear Integer Arithmetic Theory Research Track Xindi Zhang Institute of Software Chinese Academy of Science, Bohan Li Institute of Software Chinese Academy of Science, Shaowei Cai Institute of Software at Chinese Academy of Sciences | ||
11:30 15mTalk | Uncover the Premeditated Attacks: Detecting Exploitable Reentrancy Vulnerabilities by Identifying Attacker Contracts Research Track Shuo Yang Sun Yat-sen University, Jiachi Chen Sun Yat-sen University, Mingyuan Huang Sun Yat-Sen University, Zibin Zheng Sun Yat-sen University, Yuan Huang School of Data and Computer Science, Sun Yat-sen University, Guangzhou, China | ||
11:45 15mTalk | Practical Non-Intrusive GUI Exploration Testing with Visual-based Robotic Arms Research Track Shengcheng Yu Nanjing University, Chunrong Fang Nanjing University, Mingzhe Du Nanjing University, Yuchen Ling Nanjing University, Zhenyu Chen Nanjing University, Zhendong Su ETH Zurich | ||
12:00 15mTalk | Dynamic Inference of Likely Symbolic Tensor Shapes in Python Machine Learning Programs Software Engineering in Practice Pre-print | ||
12:15 7mTalk | Mutation Analysis for Evaluating Code Translation Journal-first Papers Giovani Guizzo Brick Abode, Jie M. Zhang King's College London, Federica Sarro University College London, Mark Harman Meta Platforms, Inc. and UCL, Christoph Treude Singapore Management University | ||
12:22 7mTalk | Generalized Coverage Criteria for Combinatorial Sequence Testing Journal-first Papers Achiya Elyasaf Ben-Gurion University of the Negev, Eitan Farchi IBM Haifa Research Lab, Oded Margalit Ben-Gurion University of the Negev, Gera Weiss Ben-Gurion University of the Negev, Yeshayahu Weiss Ben-Gurion University of the Negev Link to publication DOI | ||