Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototyping
This program is tentative and subject to change.
Multimodal Large Language Models (MLLMs) have demonstrated remarkable performance on the design-to-code task, i.e., generating UI code from UI mock-ups. However, existing benchmarks only contain static web pages for evaluation and ignore the dynamic interaction, limiting the practicality, usability and user engagement of the generated webpages.
To bridge these gaps, we present the first systematic investigation of MLLMs in generating interactive webpages. Specifically, we formulate the Interaction-to-Code task and establish the Interaction2Code benchmark, encompassing 127 unique webpages and 374 distinct interactions across 15 webpage types and 31 interaction categories. Through comprehensive experiments utilizing state-of-the-art (SOTA) MLLMs, evaluated via both automatic metrics and human assessments, we identify four critical limitations of MLLM on Interaction-to-Code task: (1) inadequate generation of interaction compared with full page, (2) prone to ten types of failure, (3) poor performance on visually subtle interactions, and (4) insufficient undestanding on interaction when limited to single-modality visual descriptions. To address these limitations, we propose four enhancement strategies: Interactive Element Highlighting, Failure-aware Prompting (FAP), Visual Saliency Enhancement, and Visual-Textual Descriptions Combination, all aiming at improving MLLMs’ performance on the Interaction-to-Code task. Our data and code are available in https://anonymous.4open.science/r/Interaction2Code-0E7C.
This program is tentative and subject to change.
Mon 17 NovDisplayed time zone: Seoul change
14:00 - 15:30 | |||
14:00 10mTalk | Why AI Agents Still Need You: Findings from Developer-Agent Collaborations in the Wild Research Papers Aayush Kumar Microsoft, Yasharth Bajpai Microsoft, Sumit Gulwani Microsoft, Gustavo Soares Microsoft, Emerson Murphy-Hill Microsoft | ||
14:10 10mTalk | The Cost of Downgrading Build Systems: A Case Study of Kubernetes Research Papers Gareema Ranjan University of Waterloo, Mahmoud Alfadel University of Calgary, Gengyi Sun University of Waterloo, Shane McIntosh University of Waterloo Pre-print | ||
14:20 10mTalk | Democratizing the Cryptocurrency Ecosystem by Just-In-Time Transformation of Mining Programs Research Papers Wei Liu Nanjing University, Zhenhua Li Tsinghua University, Feng Qian University of Southern California, Feiyu Jin Tsinghua University, Hao Lin Tsinghua University, Yannan Zheng Ant Group, Bo Xiao Ant Group, Xiaokang Qin Ant Group, Tianyin Xu University of Illinois at Urbana-Champaign | ||
14:30 10mTalk | Advancing Automated Ethical Profiling in SE: a Zero-Shot Evaluation of LLM Reasoning Research Papers Patrizio Migliarini University of L'Aquila, Italy, Mashal Afzal Memon University of L’Aquila, Italy, Marco Autili University of L'Aquila, Italy, Paola Inverardi Gran Sasso Science Institute Pre-print | ||
14:40 10mTalk | The Impact of the COVID-19 Pandemic on Women’s Contribution to Public Code Journal-First Track Annalí Casanueva Ifo Institute, Big Data Junior Research Group, Munich, Germany, Davide Rossi University of Bologna, Théo Zimmermann Télécom Paris, Polytechnic Institute of Paris, Stefano Zacchiroli LTCI, Télécom Paris, Institut Polytechnique de Paris, Palaiseau, France Link to publication DOI Pre-print | ||
14:50 10mTalk | Understanding Feature Request Practice on GitHub via a Large-Scale Empirical Study Research Papers Jiajun Li Nanjing University of Aeronautics and Astronautics, Wenhua Yang Nanjing University of Aeronautics and Astronautics, Minxue Pan Nanjing University, Yu Zhou Nanjing University of Aeronautics and Astronautics | ||
15:00 10mTalk | Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototyping Research Papers Jingyu Xiao The Chinese University of Hong Kong, Yuxuan Wan The Chinese University of Hong Kong, Yintong Huo Singapore Management University, Singapore, Zixin Wang The Chinese University of Hong Kong, Xinyi Xu The Chinese University of Hong Kong, Wenxuan Wang Hong Kong University of Science and Technology, Zhiyao Xu Tsinghua University, Yuhang Wang Southwest University, Michael Lyu The Chinese University of Hong Kong | ||
15:10 10mTalk | Engineering Digital Systems for Humanity: a Research Roadmap Journal-First Track Marco Autili University of L'Aquila, Italy, Martina De Sanctis Gran Sasso Science Institute, Paola Inverardi Gran Sasso Science Institute, Patrizio Pelliccione Gran Sasso Science Institute, L'Aquila, Italy | ||
15:20 10mTalk | Multi-dimensional Assessment of CrowdSourced Testing Reports via LLMs Research Papers Yue Wang NanJing University, Yuan Zhao Laboratory of Data Intelligence and Interdisciplinary Innovation, Nanjing University, Shengcheng Yu Technical University of Munich, Zhenyu Chen Nanjing University | ||