GUI-ReRank: Enhancing GUI Retrieval with Multi-Modal LLM-based Reranking
This program is tentative and subject to change.
Graphical User Interface (GUI) prototyping is a fundamental component in the development of modern interactive systems, which are now ubiquitous across diverse application domains. GUI prototypes play a critical role in requirements elicitation by enabling stakeholders to visualize, assess, and refine system concepts collaboratively. Moreover, prototypes serve as effective tools for early testing, iterative evaluation, and validation of design ideas with both end users and development teams. Despite these advantages, the process of constructing GUI prototypes remains resource-intensive and time-consuming, frequently demanding substantial effort and expertise. Recent research has sought to alleviate this burden through natural language (NL)-based GUI retrieval approaches, which typically rely on embedding-based retrieval or tailored ranking models for specific GUI repositories. However, these methods often suffer from limited retrieval performance and struggle to generalize across arbitrary GUI datasets. In this work, we present GUI-ReRank, a novel framework that integrates rapid embedding-based constrained retrieval models with highly effective multi-modal (M)LLM-based reranking techniques. GUI-ReRank further introduces a fully customizable GUI repository annotation and embedding pipeline, enabling users to effortlessly make their own GUI repositories searchable, which allows for rapid discovery of relevant GUIs for inspiration or seamless integration into customized LLM-based retrieval-augmented generation (RAG) workflows. We evaluated our approach on an established NL-based GUI retrieval benchmark, demonstrating that GUI-ReRank significantly outperforms state-of-the-art (SOTA) tailored Learning-to-Rank (LTR) models in both retrieval accuracy and generalizability. Additionally, we conducted a comprehensive cost and efficiency analysis of employing MLLMs for reranking, providing valuable insights regarding the trade-offs between retrieval effectiveness and computational resources. Video presentation of GUI-ReRank available at: https://youtu.be/_7x9UCh82ug
This program is tentative and subject to change.
Tue 18 NovDisplayed time zone: Seoul change
15:00 - 18:00 | |||
15:00 3hDemonstration | APIDA-Chat: Structured Synthesis of API Search Dialogues to Bootstrap Conversational Agents Tool Demonstration Track | ||
15:00 3hDemonstration | PROXiFY: A Bytecode Analysis Tool for Detecting and Classifying Proxy Contracts in Ethereum Smart Contracts Tool Demonstration Track Ilham Qasse Reykjavik University, Mohammad Hamdaqa Polytechnique Montreal, Björn Þór Jónsson Reykjavik University | ||
15:00 3hDemonstration | DeepTx: Real-Time Transaction Risk Analysis via Multi-Modal Features and LLM Reasoning Tool Demonstration Track Yixuan Liu Nanyang Technological University, Xinlei Li Nanyang Technological University, Yi Li Nanyang Technological University Pre-print | ||
15:00 3hDemonstration | WIBE: Watermarks for generated Images - Benchmarking & Evaluation Tool Demonstration Track Aleksey Yakushev ISP RAS, Aleksandr Akimenkov ISP RAS, Khaled Abud MSU AI Institute, Dmitry Obydenkov ISP RAS, Irina Serzhenko MIPT, Kirill Aistov Huawei Research Center, Egor Kovalev MSU, Stanislav Fomin ISP RAS, Anastasia Antsiferova ISP RAS Research Center, MSU AI Institute, Kirill Lukianov ISP RAS Research Center, MIPT, Yury Markin ISP RAS | ||
15:00 3hDemonstration | EyeNav: Accessible Webpage Interaction and Testing using Eye-tracking and NLP Tool Demonstration Track Juan Diego Yepes-Parra Universidad de los Andes, Colombia, Camilo Escobar-Velásquez Universidad de los Andes, Colombia Link to publication Media Attached | ||
15:00 3hDemonstration | Quirx: A Mutation-Based Framework for Evaluating Prompt Robustness in LLM-based Software Tool Demonstration Track Souhaila Serbout University of Zurich, Zurich, Switzerland | ||
15:00 3hDemonstration | BenGQL: An Extensible Benchmarking Framework for Automated GraphQL Testing Tool Demonstration Track Media Attached | ||
15:00 3hDemonstration | evalSmarT: An LLM-Based Evaluation Framework for Smart Contract Comment Generation Tool Demonstration Track Fatou Ndiaye MBODJI SnT, University of Luxembourg, Mame Marieme Ciss SOUGOUFARA UCAD, Senegal, Wendkuuni Arzouma Marc Christian OUEDRAOGO SnT, University of Luxembourg, Alioune Diallo University of Luxembourg, Kui Liu Huawei, Jacques Klein University of Luxembourg, Tegawendé F. Bissyandé University of Luxembourg Pre-print | ||
15:00 3hDemonstration | LLMorph: Automated Metamorphic Testing of Large Language Models Tool Demonstration Track Steven Cho The University of Auckland, New Zealand, Stefano Ruberto JRC European Commission, Valerio Terragni University of Auckland | ||
15:00 3hDemonstration | TRUSTVIS: A Multi-Dimensional Trustworthiness Evaluation Framework for Large Language Models Tool Demonstration Track Ruoyu Sun University of Alberta, Canada, Da Song University of Alberta, Jiayang Song Macau University of Science and Technology, Yuheng Huang The University of Tokyo, Lei Ma The University of Tokyo & University of Alberta | ||
15:00 3hDemonstration | GUI-ReRank: Enhancing GUI Retrieval with Multi-Modal LLM-based Reranking Tool Demonstration Track Kristian Kolthoff Institute for Software and Systems Engineering, Clausthal University of Technology, Felix Kretzer human-centered systems Lab (h-lab), Karlsruhe Institute of Technology (KIT) , Christian Bartelt Institute for Software and Systems Engineering, TU Clausthal, Alexander Maedche human-centered systems Lab (h-lab), Karlsruhe Institute of Technology (KIT) , Simone Paolo Ponzetto Data and Web Science Group, University of Mannheim Pre-print Media Attached | ||
15:00 3hDemonstration | StackPlagger: A System for Identifying AI-Code Plagiarism on Stack Overflow Tool Demonstration Track Aman Swaraj Dept. of Computer Science & Engineering, Indian Institute of Technology, Roorkee, India, Harsh Goyal Indian Institute of Technology, Roorkee, Sumit Chadgal Indian Institute of Technology, Roorkee, Sandeep Kumar Dept. of Computer Science & Engineering, Indian Institute of Technology, Roorkee, India | ||
15:00 3hDemonstration | AgentDroid: A Multi-Agent Tool for Detecting Fraudulent Android Applications Tool Demonstration Track Ruwei Pan Chongqing University, Hongyu Zhang Chongqing University, Zhonghao Jiang , Ran Hou Chongqing University | ||