Accelerating OCR-Based Widget Localization for Test Automation of GUI ApplicationsVirtual
Optical character recognition (OCR) algorithms often run slow. They may take several seconds to recognize the texts on a GUI screen, which makes OCR-based widget localization in test automation unfriendly for use, especially on GPU-free computers. This paper first concludes a common type of widget text to be located in GUI testing: label text, which are short texts in widgets like buttons, menu items, and window titles. We then investigate the characteristics of texts on a GUI screen and introduce a fast GPU-independent Label Text Screening (LTS) technique to accelerate the OCR process for label text localization. The technique opens the black box of OCR engines and uses a combination of simple methods to avoid excessive text analysis on a screen as much as possible. Experiments show that, on the subject datasets, LTS reduces the average OCR-based label text localization time to a large extent. On 4k resolution GUI screens, it keeps the localization time below 0.5 seconds for over about 60% of cases without GPU support on a normal laptop computer. In contrast, the existing CPU-based approaches built on popular OCR engines Tesseract, PaddleOCR, and EasyOCR usually need over 2 seconds to achieve the same goal on the same platform. Even with GPU acceleration, they can hardly keep the analysis time in 1 second. We believe the proposed approach would be helpful for implementing OCR-based test automation tools.
Wed 12 OctDisplayed time zone: Eastern Time (US & Canada) change
13:30 - 15:30 | Technical Session 13 - Application DomainsNIER Track / Research Papers / Journal-first Papers at Ballroom C East Chair(s): Andrea Stocco Università della Svizzera italiana (USI) | ||
13:30 20mResearch paper | A Hybrid Approach for Inference between Behavioral Exception API Documentation and Implementations, and Its Applications Research Papers Hoan Anh Nguyen Amazon, Hung Phan , Samantha Syeda Khairunnesa Bradley University, Son Nguyen The University of Texas at Dallas, Aashish Yadavally University of Texas at Dallas, Shaohua Wang New Jersey Institute of Technology, Hridesh Rajan Iowa State University, Tien N. Nguyen University of Texas at Dallas | ||
13:50 10mVision and Emerging Results | Property-Based Automated Repair of DeFi Protocols NIER Track Palina Tolmach Nanyang Technological University, Singapore, Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR), Singapore, Yi Li Nanyang Technological University, Shang-Wei Lin Nanyang Technological University Pre-print | ||
14:00 20mResearch paper | Automatically Detecting Visual Bugs in HTML5 <canvas> Games Research Papers Finlay Macklon University of Alberta, Mohammad Reza Taesiri University of Alberta, Markos Viggiato University of Alberta, Stefan Antoszko University of Alberta, Natalia Romanova Prodigy Education, Dale Paas Prodigy Education, Cor-Paul Bezemer University of Alberta | ||
14:20 10mVision and Emerging Results | Reflecting on Recurring Failures in IoT Development NIER Track | ||
14:30 20mResearch paper | Empirical Study of System Resources Abused by IoT AttackersVirtual Research Papers Zijing Yin Tsinghua University, Yiwen Xu Tsinghua University, Chijin Zhou Tsinghua University, Yu Jiang Tsinghua University | ||
14:50 20mPaper | Large-Scale Empirical Study of Inline Assembly on 7.6 Million Ethereum Smart ContractsVirtual Journal-first Papers Xiao Peng China EverBright Bank, Shuwei Song University of Electronic Science and Technology of China, Xiao Peng China EverBright Bank, Xiapu Luo Hong Kong Polytechnic University, Xiao Peng China EverBright Bank, Xiao Peng China EverBright Bank, Ting Chen University of Electronic Science and Technology of China, Jiachi Chen Sun Yat-Sen University, Tao Zhang Macau University of Science and Technology (MUST), Xiaosong Zhang University of Electronic Science and Technology of China Link to publication DOI | ||
15:10 20mResearch paper | Accelerating OCR-Based Widget Localization for Test Automation of GUI ApplicationsVirtual Research Papers Ju Qian Nanjing University of Aeronautics and Astronautics, Yingwei Ma Nanjing University of Aeronautics and Astronautics, Chenghao Lin Nanjing University of Aeronautics and Astronautics, Lin Chen Nanjing University |