Prompts Are Programs Too! Understanding How Developers Build Software Containing Prompts
The introduction of generative pre-trained models, like GPT-4, has introduced a phenomenon known as prompt engineering, whereby model users repeatedly write and revise prompts while trying to achieve a task. Using these AI models for intelligent features in software applications require using APIs that are controlled through developer-written prompts. These prompts have powered AI experiences in popular software products, potentially reaching millions of users. Despite the growing impact of prompt-powered software, little is known about its development process and its relationship to programming. In this work, we argue that some forms of prompts are programs, and that the development of prompts is a distinct phenomenon in programming. We refer to this phenomenon prompt programming. To this end, we develop an understanding of prompt programming using Straussian grounded theory through interviews with 20 developers engaged in prompt development across a variety of contexts, models, domains, and prompt complexities.
Through this study, we contribute 14 observations about prompt programming. For example, rather than building mental models of code, prompt programmers develop mental models of the FM’s behavior on the prompt and its unique qualities by interacting with the model. While prior research has shown that experts have well-formed mental models, we find that prompt programmers who have written dozens of prompts still struggle to develop reliable mental models, causing a rapid and unsystematic development process. Taken together, our observations indicate that prompt programming is significantly different from traditional software development, motivating the creation of tools to support prompt programming. Our findings have implications for software engineering practitioners, educators, and researchers.
Wed 25 JunDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
11:00 - 12:30 | LLM and PromptIndustry Papers / Ideas, Visions and Reflections / Research Papers / Journal First at Cosmos 3B Chair(s): Giuseppe Scanniello University of Salerno | ||
11:00 20mTalk | On Inter-dataset Code Duplication and Data Leakage in Large Language Models Journal First José Antonio Hernández López Linköping University, Boqi Chen McGill University, Mootez Saad Dalhousie University, Tushar Sharma Dalhousie University, Daniel Varro Linköping University / McGill University | ||
11:20 20mTalk | LLM App Squatting and Cloning Industry Papers Yinglin Xie Huazhong University of Science and Technology, Xinyi Hou Huazhong University of Science and Technology, Yanjie Zhao Huazhong University of Science and Technology, Kai Chen Huazhong University of Science and Technology, Haoyu Wang Huazhong University of Science and Technology | ||
11:40 10mTalk | Predictive Prompt Analysis Ideas, Visions and Reflections | ||
11:50 20mTalk | From Prompts to Templates: A Systematic Prompt Template Analysis for Real-world LLMapps Industry Papers Yuetian Mao Technical University of Munich, Junjie He Technical University of Munich, Chunyang Chen TU Munich | ||
12:10 20mTalk | Prompts Are Programs Too! Understanding How Developers Build Software Containing Prompts Research Papers Jenny T. Liang Carnegie Mellon University, Melissa Lin Carnegie Mellon University, Nikitha Rao Carnegie Mellon University, Brad A. Myers Carnegie Mellon University DOI | ||
Cosmos 3B is the second room in the Cosmos 3 wing.
When facing the main Cosmos Hall, access to the Cosmos 3 wing is on the left, close to the stairs. The area is accessed through a large door with the number “3”, which will stay open during the event.