ESEIW 2024
Sun 20 - Fri 25 October 2024 Barcelona, Spain

Large language models have shown good potential in supporting software development tasks. This is why more and more developers turn to LLMs (e.g. ChatGPT) to support them in fixing their buggy code. While this can save time and effort, many companies prohibit it due to strict code sharing policies. To address this, company can run open-source LLMs locally. But until now there is not much research evaluating the performance of open-source large language models in debugging. This work is a preliminary evaluation of the capabilities of open-source LLMs in fixing buggy code. The evaluation covers five open-source large language models and uses the Benchmark DebugBench which includes more than 4000 buggy code instances written in Python, Java and C++. Open-source LLMs achieved scores ranging from 43.9% to 65.7% with DeepSeek-Coder achieving the best score for all three programming languages.

Fri 25 Oct

Displayed time zone: Brussels, Copenhagen, Madrid, Paris change

14:00 - 15:30
Large language models in software engineering IIESEM Emerging Results, Vision and Reflection Papers Track / ESEM IGC at Telensenyament (B3 Building - 1st Floor)
Chair(s): Claudio Di Sipio University of l'Aquila
14:00
15m
Vision and Emerging Results
Debugging with Open-Source Large Language Models: An Evaluation
ESEM Emerging Results, Vision and Reflection Papers Track
Yacine Majdoub IResCoMath Lab, University of Gabes, Eya Ben Charrada IResCoMath Lab, University of Gabes
Link to publication DOI Pre-print
14:15
15m
Vision and Emerging Results
Multi-language Software Development in the LLM Era: Insights from Practitioners’ Conversations with ChatGPT
ESEM Emerging Results, Vision and Reflection Papers Track
Lucas Almeida Aguiar State University of Ceará, Matheus Paixao State University of Ceará, Rafael Carmo Federal University of Ceará, Edson Soares Instituto Atlantico & State University of Ceara (UECE), Antonio Leal State University of Ceará, Matheus Freitas State University of Ceará, Eliakim Gama State University of Ceará
14:30
15m
Vision and Emerging Results
Exploring LLM-Driven Explanations for Quantum Algorithms
ESEM Emerging Results, Vision and Reflection Papers Track
Giordano d'Aloisio University of L'Aquila, Sophie Fortz King's College London, Carol Hanna University College London, Daniel Fortunato INESC-ID, University of Porto, Avner Bensoussan King's College London, Eñaut Mendiluze Usandizaga Simula Research Laboratory, Norway, Federica Sarro University College London
Pre-print
14:45
15m
Industry talk
Beyond Words: On Large Language Models Actionability in Mission-Critical Risk Analysis
ESEM IGC
Matteo Esposito University of Oulu, Francesco Palagiano Multitel di Lerede Alessandro & C. s.a.s., Valentina Lenarduzzi University of Oulu, Davide Taibi University of Oulu
Pre-print
15:00
15m
Vision and Emerging Results
Detecting Code Smells using ChatGPT: Initial Insights
ESEM Emerging Results, Vision and Reflection Papers Track
Luciana L. Silva Federal University of Minas Gerais, Janio R. Silva IFMG, João Eduardo Montandon Universidade Federal de Minas Gerais (UFMG), Marcus Andrade IFMG, Marco Tulio Valente Federal University of Minas Gerais, Brazil
15:15
15m
Industry talk
ChatGPT’s Potential in Cryptography Misuse Detection: A Comparative Analysis with Static Analysis Tools
ESEM IGC
Ehsan Firouzi TU Clausthal, Mohammad Ghafari TU Clausthal, Mike Ebrahimi CUBE