ICSE 2024
Fri 12 - Sun 21 April 2024 Lisbon, Portugal

With their exceptional natural language processing capabilities, tools based on Large Language Models (LLMs) like ChatGPT and Co-Pilot have swiftly become indispensable resources in the software developer’s toolkit. While recent studies suggest the potential productivity gains these tools can unlock, users still encounter drawbacks, such as generic or incorrect answers. Additionally, the pursuit of improved responses often leads to extensive prompt engineering efforts, diverting valuable time from writing code that delivers actual value. To address these challenges, a new breed of tools, built atop LLMs, is emerging. These tools aim to mitigate drawbacks by employing techniques like fine-tuning or enriching user prompts with contextualized information.

In this paper, we delve into the lessons learned by a software development team venturing into the creation of such a contextualized LLM-based application, using retrieval-based techniques, called CodeBuddy. Over a four-month period, the team, despite lacking prior professional experience in LLM-based applications, built the product from scratch. Following the initial product release, we engaged with the development team responsible for the code generative components. Through interviews and analysis of the application’s issue tracker, we uncover various intriguing challenges that teams working on LLM-based applications might encounter. For instance, we found three main group of lessons: LLM-based lessons, User-based lessons, and Technical lessons. By understanding these lessons, software development teams could become better prepared to build LLM-based applications.

Fri 19 Apr

Displayed time zone: Lisbon change

16:00 - 17:30
16:00
15m
Talk
Predicting Performance and Accuracy of Mixed-Precision Programs for Precision Tuning
Research Track
Yutong Wang University of California, Davis, Cindy Rubio-González University of California at Davis
16:15
15m
Talk
A Synthesis of Green Architectural Tactics for ML-Enabled Systems
Software Engineering in Society
Heli Järvenpää Vrije Universiteit Amsterdam, Patricia Lago Vrije Universiteit Amsterdam, Justus Bogner Vrije Universiteit Amsterdam, Grace Lewis Carnegie Mellon Software Engineering Institute, Henry Muccini University of L'Aquila, Italy, Ipek Ozkaya Carnegie Mellon University
Pre-print
16:30
15m
Talk
Greening Large Language Models of Code
Software Engineering in Society
Jieke Shi Singapore Management University, Zhou Yang Singapore Management University, Hong Jin Kang UCLA, Bowen Xu North Carolina State University, Junda He Singapore Management University, David Lo Singapore Management University
Pre-print Media Attached
16:45
15m
Talk
Lessons from Building CodeBuddy: A Contextualized AI Coding Assistant
Software Engineering in Practice
Gustavo Pinto Federal University of Pará (UFPA) and Zup Innovation, Cleidson de Souza Federal University of Pará Belém, João Batista Cordeiro Neto Federal University of Santa Catarina and Zup Innovation, Alberto de Souza Zup Innovation, Tarcísio Gotto Zup Innovation, Edward Monteiro StackSpot
17:00
15m
Talk
CodeFuse-13B: A Pretrained Multi-lingual Code Large Language Model
Software Engineering in Practice
Peng Di Ant Group, Jianguo Li Ant Group, Hang Yu Ant Group, Wei Jiang Ant Group
17:15
7m
Talk
Breaking the Silence: the Threats of Using LLMs in Software Engineering
New Ideas and Emerging Results
June Sallou Delft University of Technology, Thomas Durieux TU Delft, Annibale Panichella Delft University of Technology
Pre-print