ESEIW 2025
Sun 28 September - Fri 3 October 2025
Fri 3 Oct 2025 11:20 - 11:38 at Kaiulani I - Systematic Reviews and Evidence-Based SE Chair(s): Nauman Bin Ali

Systematic Literature Reviews (SLRs) and Mapping Studies (MSs) are an established feature of academic Software Engineering literature. But for new, exciting advances in the area there is a tension between supporting fellow academics with reviews that provide a roadmap of the area/timely synthesis of the available evidence, and the maturity of the literature relied on for those reviews. The increasingly prevalent use of preprint databases such as arXiv in Computer Science, for the immediate publication of results, suggests that this tension deserves evaluation and this paper does so through the lens of LLMs4SE: Large Language Models for Software Engineering. Specifically, this paper reports on the initial wave of SLRs and MSs covering LLMs4SE. It directs its analysis at identifying the proportion of arXiv preprints the existing surveys employ, identifying any additional quality assurance measures applied to those preprints, and tracing any subsequent peer-reviewed publications of those preprints. Our initial review found a small number of surveys in the area, congruent with LLMs recent growth as a significant tool in software engineering. The reviews that do exist rely on a significant proportion of pre-prints, and meaningful quality criteria are rarely applied to mitigate against this reliance. We note that although a significant proportion of the preprints are subsequently published in peer-reviewed fora, a significant number are not. In cases where the preprints have been subsequently published in peer-reviewed fora, consistency between the assertions made in the review and the peer-reviewed article was high.

Fri 3 Oct

Displayed time zone: Hawaii change

11:20 - 12:50
Systematic Reviews and Evidence-Based SEESEM - Technical Track / ESEM - Registered Reports Track / ESEM - Emerging Results and Vision Track / at Kaiulani I
Chair(s): Nauman Bin Ali Blekinge Institute of Technology
11:20
18m
Talk
A Preliminary Assessment of SLR’s Reliance on Preprints, in the area of LLMs4SE
ESEM - Emerging Results and Vision Track
Sarah Buckley University of Limerick, Abdul Razzaq Computer Science and Information Systems, University of Limerick, Michael English Computer Science and Information Systems, University of Limerick
11:38
18m
Talk
Investigating the Use of LLMs for Evidence Briefings Generation in Software Engineering
ESEM - Registered Reports Track
Mauro Marcelino Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Marcos Alves FITec Technological Innovations, Bianca Trinkenreich Colorado State University, Bruno Cartaxo IFPE, Sérgio Soares Universidade Federal de Pernambuco, Simone Barbosa PUC-Rio, Marcos Kalinowski Pontifical Catholic University of Rio de Janeiro (PUC-Rio)
11:56
18m
Talk
Another Systematic Review? A Critical Analysis of Systematic Literature Reviews on Agile Effort and Cost Estimation
ESEM - Technical Track
Henry Edison Blekinge Institute of Technology, Nauman Ali Blekinge Institute of Technology
12:14
18m
Talk
Assessing diversity in creating seed set for snowballing search for systematic literature review in software engineering
ESEM - Technical Track
Katia Romero Felizardo Federal University of Technology - Paraná (UTFPR), Francisco Carlos M. Souza Universidade Tecnológica Federal do Paraná (UTFPR), Alinne C. Corrêa Souza Universidade Tecnológica Federal do Paraná (UTFPR), Igor Steinmacher RESHAPE LAB, Northern Arizona University, USA, Marco Gerosa Northern Arizona University
12:32
18m
Talk
SESR-Eval: Dataset to Evaluate LLMs in the Screening Process of Systematic Reviews
ESEM - Technical Track
Aleksi Huotala University of Helsinki, Miikka Kuutila Dalhousie University, Mika Mäntylä University of Helsinki and University of Oulu
Pre-print