On the difficulties of conducting and replicating systematic literature reviews studies using LLMs in software engineering
The software engineering community has adopted systematic literature reviews (SLRs) to summarize the state-of-the-art in specific research topics. SLRs offer benefits such as synthesizing evidence from diverse studies to generate auditable results following a reproducible approach and identifying research gaps for future exploration. However, the process is effort-intensive, prone to errors, and lays various challenges during their conduction. To overcome some of these issues, there is a growing belief that large language models (LLMs) can support systematic literature reviews. While the literature has shown promising results in social sciences, more evidence of its accuracy is needed in technical fields like software engineering. In this context, studies and replications are essential in verifying the benefits and drawbacks of applying large language models in systematic literature reviews. This paper discusses the difficulties in conducting and replicating studies that adopt large language models to support systematic literature in software engineering. As an implication, we identified the challenges of adopting LLM in SLRs and offered a list of open issues for future research.
Sat 3 MayDisplayed time zone: Eastern Time (US & Canada) change
14:00 - 15:30 | |||
14:00 50mKeynote | Empirical Software Engineering in the Public Sector WSESE Arie van Deursen TU Delft Media Attached | ||
14:50 12mTalk | The Role of Paper-Type in Systematic Mapping Studies in Software Engineering WSESE | ||
15:02 12mTalk | On the difficulties of conducting and replicating systematic literature reviews studies using LLMs in software engineering WSESE Katia Romero Felizardo NAU RESHAPE LAB, Anderson Deizepe UTFPR-CP, Daniel Coutinho Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Genildo Gomes da Silva Junior , Maria Alcimar Costa Meireles UFAM - Federal University of Amazonas, Marco Gerosa Northern Arizona University, Igor Steinmacher NAU RESHAPE LAB | ||
15:14 16mLive Q&A | Keynote & Secondary Studies: Discussion WSESE |