From Prompts to Templates: A Systematic Prompt Template Analysis for Real-world LLMapps
Large Language Models (LLMs) have revolutionized human-AI interaction by enabling intuitive task execution through natural language prompts. Despite their potential, designing effective prompts remains a significant challenge, as small variations in structure or wording can result in substantial differences in output. To address these challenges, LLM-powered applications (LLMapps) rely on prompt templates to simplify interactions, enhance usability, and support specialized tasks such as document analysis, creative content generation, and code synthesis. However, current practices heavily depend on individual expertise and iterative trial-and-error processes, underscoring the need for systematic methods to optimize prompt template design in LLMapps. This paper presents a comprehensive analysis of prompt templates in practical LLMapps. We construct a dataset of real-world templates from open-source LLMapps, including those from leading companies like Uber and Microsoft. Through a combination of LLM-driven analysis and human review, we categorize template components and placeholders, analyze their distributions, and identify frequent co-occurrence patterns. Additionally, we evaluate the impact of identified patterns on LLMs’ instruction-following performance through sample testing. Our findings provide practical insights on prompt template design for developers, supporting the broader adoption and optimization of LLMapps in industrial settings.
Wed 25 JunDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
11:00 - 12:30 | LLM and PromptIndustry Papers / Ideas, Visions and Reflections / Research Papers / Journal First at Cosmos 3B Chair(s): Giuseppe Scanniello University of Salerno | ||
11:00 20mTalk | On Inter-dataset Code Duplication and Data Leakage in Large Language Models Journal First José Antonio Hernández López Linköping University, Boqi Chen McGill University, Mootez Saad Dalhousie University, Tushar Sharma Dalhousie University, Daniel Varro Linköping University / McGill University | ||
11:20 20mTalk | LLM App Squatting and Cloning Industry Papers Yinglin Xie Huazhong University of Science and Technology, Xinyi Hou Huazhong University of Science and Technology, Yanjie Zhao Huazhong University of Science and Technology, Kai Chen Huazhong University of Science and Technology, Haoyu Wang Huazhong University of Science and Technology | ||
11:40 10mTalk | Predictive Prompt Analysis Ideas, Visions and Reflections | ||
11:50 20mTalk | From Prompts to Templates: A Systematic Prompt Template Analysis for Real-world LLMapps Industry Papers Yuetian Mao Technical University of Munich, Junjie He Technical University of Munich, Chunyang Chen TU Munich | ||
12:10 20mTalk | Prompts Are Programs Too! Understanding How Developers Build Software Containing Prompts Research Papers Jenny T. Liang Carnegie Mellon University, Melissa Lin Carnegie Mellon University, Nikitha Rao Carnegie Mellon University, Brad A. Myers Carnegie Mellon University DOI |
Cosmos 3B is the second room in the Cosmos 3 wing.
When facing the main Cosmos Hall, access to the Cosmos 3 wing is on the left, close to the stairs. The area is accessed through a large door with the number “3”, which will stay open during the event.