MLLM-Based UI2Code Automation Guided by UI Layout Information
Converting user interfaces into code (UI2Code) is a crucial step in website development, which is time-consuming and labor-intensive. The automation of UI2Code is essential to streamline this task, beneficial for improving the development efficiency. There exist deep learning-based methods for the task; however, they heavily rely on a large amount of labeled training data and struggle with generalizing to real-world, unseen web page designs. The advent of Multimodal Large Language Models (MLLMs) presents potential for alleviating the issue, but they are difficult to comprehend the complex layouts in UIs and generate the accurate code with layout preserved. To address these issues, we propose LayoutCoder, a novel MLLM-based framework generating UI code from real-world webpage images, which includes three key modules: (1) Element Relation Construction, which aims at capturing UI layout by identifying and grouping components with similar structures; (2) UI Layout Parsing, which aims at generating UI layout trees for guiding the subsequent code generation process; and (3) Layout-Guided Code Fusion, which aims at producing the accurate code with layout preserved. For evaluation, we build a new benchmark dataset which involves 350 real-world websites named Snap2Code, divided into seen and unseen parts for mitigating the data leakage issue, besides the popular dataset Design2Code. Extensive evaluation shows the superior performance of LayoutCoder over the state-of-the-art approaches. Compared with the best-performing baseline, LayoutCoder improves 10.14% in the BLEU score and 3.95% in the CLIP score on average across all datasets.
Thu 26 JunDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
14:00 - 15:15 | |||
14:00 25mTalk | Are Autonomous Web Agents good testers? Research Papers Antoine Chevrot Smartesting, Alexandre Vernotte Smartesting, Jean-Rémy Falleri Univ. Bordeaux, CNRS, Bordeaux INP, LaBRI, UMR 5800, Institut Universitaire de France, Xavier Blanc Université de Bordeaux, Bruno Legeard Université de Bourgogne Franche-Comté and Smartesting, Aymeric Cretin Smartesting DOI | ||
14:25 25mTalk | MLLM-Based UI2Code Automation Guided by UI Layout Information Research Papers Fan Wu Harbin Institute of Technology, Shenzhen, Cuiyun Gao Harbin Institute of Technology, Shuqing Li The Chinese University of Hong Kong, Xin-Cheng Wen Harbin Institute of Technology, Qing Liao Harbin Institute of Technology DOI Pre-print | ||
14:50 25mTalk | GUIPilot: A Consistency-based Mobile GUI Testing Approach for Detecting Application-specific Bugs Research Papers Ruofan Liu Shanghai Jiao Tong University; National University of Singapore, Xiwen Teoh National University of Singapore, Yun Lin Shanghai Jiao Tong University, Guanjie Chen Shanghai Jiao Tong University, Ruofei Ren Shanghai Jiao Tong University, Denys Poshyvanyk William & Mary, Jin Song Dong National University of Singapore Link to publication DOI File Attached |
Cosmos 3C is the third room in the Cosmos 3 wing.
When facing the main Cosmos Hall, access to the Cosmos 3 wing is on the left, close to the stairs. The area is accessed through a large door with the number “3”, which will stay open during the event.