DeclarUI: Bridging Design and Development with Automated Declarative UI Code Generation
Declarative UI frameworks have gained widespread adoption in mobile app development, offering benefits such as improved code readability and easier maintenance. Despite these advantages, the process of translating UI designs into functional code remains challenging and time-consuming. Recent advancements in multimodal large language models (MLLMs) have shown promise in directly generating mobile app code from user interface (UI) designs. However, the direct application of MLLMs to this task is limited by challenges in accurately recognizing UI components and comprehensively capturing interaction logic.
To address these challenges, we propose DeclarUI, an automated approach that synergizes computer vision (CV), MLLMs, and iterative compiler-driven optimization to generate and refine declarative UI code from designs. DeclarUI enhances visual fidelity, functional completeness, and code quality through precise component segmentation, Page Transition Graphs (PTGs) for modeling complex inter-page relationships, and iterative optimization. In our evaluation, DeclarUI outperforms baselines on React Native, a widely adopted declarative UI framework, achieving a 96.8% PTG coverage rate and a 98% compilation success rate. Notably, DeclarUI demonstrates significant improvements over state-of-the-art MLLMs, with a 123% increase in PTG coverage rate, up to 55% enhancement in visual similarity scores, and a 29% boost in compilation success rate. We further demonstrate DeclarUI’s generalizability through successful applications to Flutter and ArkUI frameworks. User studies with professional developers confirm that DeclarUI’s generated code meets industrial-grade standards in code availability, modification time, readability, and maintainability. By streamlining app development, improving efficiency, and fostering designer-developer collaboration, DeclarUI offers a practical solution to the persistent challenges in mobile UI development.
Mon 23 JunDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
16:00 - 17:50 | Code Generation 1Industry Papers / Demonstrations / Research Papers / Journal First at Cosmos 3C Chair(s): Zhongxin Liu Zhejiang University | ||
16:00 20mTalk | How Do Programming Students Use Generative AI? Research Papers DOI Pre-print | ||
16:20 20mTalk | Towards Mitigating API Hallucination in Code Generated by LLMs with Hierarchical Dependency Aware Industry Papers Yujia Chen Harbin Institute of Technology, Shenzhen, Mingyu Chen Harbin Institute of Technology, Shenzhen, Cuiyun Gao Harbin Institute of Technology, Shenzhen, Zhihan Jiang Huawei Cloud Computing Technologies Co., Ltd., Zhongqi Li Huawei Cloud Computing Technologies Co., Ltd., Yuchi Ma Huawei Cloud Computing Technologies | ||
16:40 10mTalk | CoSEFA: An LLM-Based Programming Assistant for Secure Code Generation via Supervised Co-Decoding Demonstrations Xuan He Chongqing University, Dong Li Chongqing University, Hao Wen CloudWalk Technology Co., Ltd, Yueheng Zhu Chongqing University, Chao Liu Chongqing University, Meng Yan Chongqing University, Hongyu Zhang Chongqing University | ||
16:50 20mTalk | DeclarUI: Bridging Design and Development with Automated Declarative UI Code Generation Research Papers Ting Zhou Huazhong University of Science and Technology, Yanjie Zhao Huazhong University of Science and Technology, Xinyi Hou Huazhong University of Science and Technology, Xiaoyu Sun Australian National University, Australia, Kai Chen Huazhong University of Science and Technology, Haoyu Wang Huazhong University of Science and Technology DOI | ||
17:10 20mTalk | RAG or Fine-tuning? A Comparative Study on LCMs-based Code Completion in Industry Industry Papers Chaozheng Wang The Chinese University of Hong Kong, Zezhou Yang Tencent Inc., Shuzheng Gao Chinese University of Hong Kong, Cuiyun Gao Harbin Institute of Technology, Shenzhen, Ting Peng Tencent Inc., Hailiang Huang Tencent Inc., Yuetang Deng Tencent, Michael Lyu Chinese University of Hong Kong | ||
17:30 20mTalk | Automated Code Editing with Search-Generate-Modify Journal First Changshu Liu Columbia University, Pelin Cetin Columbia University, Yogesh Patodia Columbia University, Baishakhi Ray Columbia University, Saikat Chakraborty Microsoft Research, Yangruibo Ding Columbia University Pre-print Media Attached File Attached |
Cosmos 3C is the third room in the Cosmos 3 wing.
When facing the main Cosmos Hall, access to the Cosmos 3 wing is on the left, close to the stairs. The area is accessed through a large door with the number “3”, which will stay open during the event.