A Preliminary Assessment of SLR’s Reliance on Preprints, in the area of LLMs4SE
Systematic Literature Reviews (SLRs) and Mapping Studies (MSs) are an established feature of academic Software Engineering literature. But for new, exciting advances in the area there is a tension between supporting fellow academics with reviews that provide a roadmap of the area/timely synthesis of the available evidence, and the maturity of the literature relied on for those reviews. The increasingly prevalent use of preprint databases such as arXiv in Computer Science, for the immediate publication of results, suggests that this tension deserves evaluation and this paper does so through the lens of LLMs4SE: Large Language Models for Software Engineering. Specifically, this paper reports on the initial wave of SLRs and MSs covering LLMs4SE. It directs its analysis at identifying the proportion of arXiv preprints the existing surveys employ, identifying any additional quality assurance measures applied to those preprints, and tracing any subsequent peer-reviewed publications of those preprints. Our initial review found a small number of surveys in the area, congruent with LLMs recent growth as a significant tool in software engineering. The reviews that do exist rely on a significant proportion of pre-prints, and meaningful quality criteria are rarely applied to mitigate against this reliance. We note that although a significant proportion of the preprints are subsequently published in peer-reviewed fora, a significant number are not. In cases where the preprints have been subsequently published in peer-reviewed fora, consistency between the assertions made in the review and the peer-reviewed article was high.
Fri 3 OctDisplayed time zone: Hawaii change
11:20 - 12:50 | Systematic Reviews and Evidence-Based SEESEM - Technical Track / ESEM - Registered Reports Track / ESEM - Emerging Results and Vision Track / at Kaiulani I Chair(s): Nauman Bin Ali Blekinge Institute of Technology | ||
11:20 18mTalk | A Preliminary Assessment of SLR’s Reliance on Preprints, in the area of LLMs4SE ESEM - Emerging Results and Vision Track Sarah Buckley University of Limerick, Abdul Razzaq Computer Science and Information Systems, University of Limerick, Michael English Computer Science and Information Systems, University of Limerick | ||
11:38 18mTalk | Investigating the Use of LLMs for Evidence Briefings Generation in Software Engineering ESEM - Registered Reports Track Mauro Marcelino Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Marcos Alves FITec Technological Innovations, Bianca Trinkenreich Colorado State University, Bruno Cartaxo IFPE, Sérgio Soares Universidade Federal de Pernambuco, Simone Barbosa PUC-Rio, Marcos Kalinowski Pontifical Catholic University of Rio de Janeiro (PUC-Rio) | ||
11:56 18mTalk | Another Systematic Review? A Critical Analysis of Systematic Literature Reviews on Agile Effort and Cost Estimation ESEM - Technical Track | ||
12:14 18mTalk | Assessing diversity in creating seed set for snowballing search for systematic literature review in software engineering ESEM - Technical Track Katia Romero Felizardo Federal University of Technology - Paraná (UTFPR), Francisco Carlos M. Souza Universidade Tecnológica Federal do Paraná (UTFPR), Alinne C. Corrêa Souza Universidade Tecnológica Federal do Paraná (UTFPR), Igor Steinmacher RESHAPE LAB, Northern Arizona University, USA, Marco Gerosa Northern Arizona University | ||
12:32 18mTalk | SESR-Eval: Dataset to Evaluate LLMs in the Screening Process of Systematic Reviews ESEM - Technical Track Aleksi Huotala University of Helsinki, Miikka Kuutila Dalhousie University, Mika Mäntylä University of Helsinki and University of Oulu Pre-print | ||