EASE 2024
Tue 18 - Fri 21 June 2024 Salerno, Italy

Context: Systematic review (SR) is a popular research method in software engineering (SE). However, conducting an SR takes an average of 67 weeks. Thus, automating any step of the SR process could reduce the effort associated with SRs. Objective: Our objective is to investigate the extent to which Large Language Models (LLMs) can accelerate title-abstract screening by (1) simplifying abstracts for human screeners, and (2) automating title-abstract screening entirely. Method: We performed an experiment where human screeners performed title-abstract screening for 20 papers with both original and simplified abstracts from a prior SR. The experiment with human screeners was reproduced by instructing GPT-3.5 and GPT-4 LLMs to perform the same screening tasks. We also studied whether different prompting techniques (Zero-shot (ZS), One-shot (OS), Few-shot (FS), and Few-shot with Chain-of-Thought (FS-CoT) prompting) improve the screening performance of LLMs. Lastly, we studied if redesigning the prompt used in the LLM reproduction of title-abstract screening leads to improved screening performance. Results: Text simplification did not increase the screeners’ screening performance, but reduced the time used in screening. Screeners’ scientific literacy skills and researcher status predict screening performance. Some LLM and prompt combinations perform as well as human screeners in the screening tasks. Our results indicate that a more recent LLM (GPT-4) is better than its predecessor LLM (GPT-3.5). Additionally, Few-shot and One-shot prompting outperforms Zero-shot prompting. Conclusion: Using LLMs for text simplification in the screening process does not significantly improve human performance. Using LLMs to automate title-abstract screening seems promising, but current LLMs are not significantly more accurate than human screeners. To recommend the use of LLMs in the screening process of SRs, more research is needed. We recommend future SR studies to publish replication packages with screening data to enable more conclusive experimenting with LLM screening.

Thu 20 Jun

Displayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change

14:00 - 15:25
Artificial Intelligence for Software EngineeringIndustry / Research Papers / Short Papers, Vision and Emerging Results at Room Capri
Chair(s): Sridhar Chimalakonda Indian Institute of Technology, Tirupati, Klaus Schmid University of Hildesheim
14:00
15m
Talk
A Performance Study of LLM-Generated Code on Leetcode
Research Papers
Tristan Coignion , Clement Quinton University of Lille, Inria, Romain Rouvoy Univ. Lille / Inria / CNRS
Pre-print
14:15
15m
Talk
How Much Logs Does My Source Code File Need? Learning to Predict the Density of Logs
Research Papers
Mohamed Amine Batoun École de Technologie Supérieure, Mohammed Sayagh ETS Montreal, University of Quebec, Ali Ouni ETS Montreal, University of Quebec
14:30
15m
Talk
The Promise and Challenges of using LLMs to Accelerate the Screening Process of Systematic Reviews
Research Papers
Aleksi Huotala University of Helsinki, Miikka Kuutila Dalhousie University, Paul Ralph Dalhousie University, Mika Mäntylä University of Helsinki and University of Oulu
Link to publication DOI Pre-print
14:45
15m
Talk
AI-enabled efficient PVM performance monitoring
Industry
Mario Veniero Independent Researcher, Davide Varriale MEDIACOM SRL
DOI
15:00
15m
Talk
Automated evaluation of game content display using deep learning
Industry
Ciprian Paduraru University of Bucharest, Marina Cernat University of Bucharest, Alin Stefanescu University of Bucharest
15:15
10m
Talk
Automated categorization of pre-trained models in software engineering: A case study with a Hugging Face dataset
Short Papers, Vision and Emerging Results
Claudio Di Sipio University of L'Aquila, Riccardo Rubei University of L'Aquila, Juri Di Rocco University of L'Aquila, Davide Di Ruscio University of L'Aquila, Phuong T. Nguyen University of L’Aquila
Pre-print