Despite advances in automated testing, manual testing remains prevalent due to the high maintenance demands associated with test script fragility—scripts often break with minor changes in application structure. Recent developments in Large Language Models (LLMs) offer a potential alternative by powering Autonomous Web Agents (AWAs) that can autonomously interact with applications. These agents may serve as Autonomous Test Agents (ATAs), potentially reducing the need for maintenance-heavy automated scripts by utilising natural language instructions similar to those used by human testers.
This paper investigates the feasibility of adapting AWAs for end-to-end test execution. We contribute with (1) an adaptation framework transforming a well-known AWA into an ATA and analysing which modules support autonomous testing, (2) a benchmark of three web applications and a suite of 100 test cases, including both passing and mutated failing cases, to evaluate ATA performance, and (3) an empirical evaluation that quantifies ATA efficacy and identifies core limitations in autonomous testing agents.
Our findings reveal that while AWAs show promising results in replicating some test scenarios autonomously, limitations in handling assertions and errors indicate that further advancements are necessary before ATAs can fully replace manual testing. This work provides insights for developing more resilient and reliable autonomous testing systems, paving the way for robust, maintenance-free test automation.
Thu 26 JunDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
14:00 - 15:15 | |||
14:00 25mTalk | Are Autonomous Web Agents good testers? Research Papers Antoine Chevrot Smartesting, Alexandre Vernotte Smartesting, Jean-Rémy Falleri Univ. Bordeaux, CNRS, Bordeaux INP, LaBRI, UMR 5800, Institut Universitaire de France, Xavier Blanc Université de Bordeaux, Bruno Legeard Université de Bourgogne Franche-Comté and Smartesting, Aymeric Cretin Smartesting DOI | ||
14:25 25mTalk | MLLM-Based UI2Code Automation Guided by UI Layout Information Research Papers Fan Wu Harbin Institute of Technology, Shenzhen, Cuiyun Gao Harbin Institute of Technology, Shuqing Li The Chinese University of Hong Kong, Xin-Cheng Wen Harbin Institute of Technology, Qing Liao Harbin Institute of Technology DOI Pre-print | ||
14:50 25mTalk | GUIPilot: A Consistency-based Mobile GUI Testing Approach for Detecting Application-specific Bugs Research Papers Ruofan Liu Shanghai Jiao Tong University; National University of Singapore, Xiwen Teoh National University of Singapore, Yun Lin Shanghai Jiao Tong University, Guanjie Chen Shanghai Jiao Tong University, Ruofei Ren Shanghai Jiao Tong University, Denys Poshyvanyk William & Mary, Jin Song Dong National University of Singapore Link to publication DOI File Attached |
Cosmos 3C is the third room in the Cosmos 3 wing.
When facing the main Cosmos Hall, access to the Cosmos 3 wing is on the left, close to the stairs. The area is accessed through a large door with the number “3”, which will stay open during the event.