Lessons from Building CodeBuddy: A Contextualized AI Coding Assistant
With their exceptional natural language processing capabilities, tools based on Large Language Models (LLMs) like ChatGPT and Co-Pilot have swiftly become indispensable resources in the software developer’s toolkit. While recent studies suggest the potential productivity gains these tools can unlock, users still encounter drawbacks, such as generic or incorrect answers. Additionally, the pursuit of improved responses often leads to extensive prompt engineering efforts, diverting valuable time from writing code that delivers actual value. To address these challenges, a new breed of tools, built atop LLMs, is emerging. These tools aim to mitigate drawbacks by employing techniques like fine-tuning or enriching user prompts with contextualized information.
In this paper, we delve into the lessons learned by a software development team venturing into the creation of such a contextualized LLM-based application, using retrieval-based techniques, called CodeBuddy. Over a four-month period, the team, despite lacking prior professional experience in LLM-based applications, built the product from scratch. Following the initial product release, we engaged with the development team responsible for the code generative components. Through interviews and analysis of the application’s issue tracker, we uncover various intriguing challenges that teams working on LLM-based applications might encounter. For instance, we found three main group of lessons: LLM-based lessons, User-based lessons, and Technical lessons. By understanding these lessons, software development teams could become better prepared to build LLM-based applications.
Fri 19 AprDisplayed time zone: Lisbon change
16:00 - 17:30 | LLM, NN and other AI technologies 7Software Engineering in Society / Software Engineering in Practice / Research Track / New Ideas and Emerging Results at Grande Auditório Chair(s): Vincent J. Hellendoorn Carnegie Mellon University | ||
16:00 15mTalk | Predicting Performance and Accuracy of Mixed-Precision Programs for Precision Tuning Research Track | ||
16:15 15mTalk | A Synthesis of Green Architectural Tactics for ML-Enabled Systems Software Engineering in Society Heli Järvenpää Vrije Universiteit Amsterdam, Patricia Lago Vrije Universiteit Amsterdam, Justus Bogner Vrije Universiteit Amsterdam, Grace Lewis Carnegie Mellon Software Engineering Institute, Henry Muccini University of L'Aquila, Italy, Ipek Ozkaya Carnegie Mellon University Pre-print | ||
16:30 15mTalk | Greening Large Language Models of Code Software Engineering in Society Jieke Shi Singapore Management University, Zhou Yang Singapore Management University, Hong Jin Kang UCLA, Bowen Xu North Carolina State University, Junda He Singapore Management University, David Lo Singapore Management University Pre-print Media Attached | ||
16:45 15mTalk | Lessons from Building CodeBuddy: A Contextualized AI Coding Assistant Software Engineering in Practice Gustavo Pinto Federal University of Pará (UFPA) and Zup Innovation, Cleidson de Souza Federal University of Pará Belém, João Batista Cordeiro Neto Federal University of Santa Catarina and Zup Innovation, Alberto de Souza Zup Innovation, Tarcísio Gotto Zup Innovation, Edward Monteiro StackSpot | ||
17:00 15mTalk | CodeFuse-13B: A Pretrained Multi-lingual Code Large Language Model Software Engineering in Practice | ||
17:15 7mTalk | Breaking the Silence: the Threats of Using LLMs in Software Engineering New Ideas and Emerging Results June Sallou Delft University of Technology, Thomas Durieux TU Delft, Annibale Panichella Delft University of Technology Pre-print |