ESEIW 2024
Sun 20 - Fri 25 October 2024 Barcelona, Spain

Experimentation is an essential method for causal inference in any empirical discipline. Crossover-design experiments are common in Software Engineering (SE) research. In these, subjects apply more than one treatment in different orders. This design increases the amount of obtained data and deals with subject variability but introduces threats to internal validity like the learning and carryover effect. Vegas et al. (2015) reviewed the state of practice for crossover designs in SE research and provided guidelines on how to address its threats during data analysis while still harnessing its benefits. In this paper, we reflect on the impact of these guidelines and review the state of analysis of crossover design experiments in SE publications between 2015 and 2024. To this end, by conducting a forward snowballing of the guidelines, we survey 136 publications reporting 67 crossover-design experiments and evaluate their data analysis against the provided guidelines. The results show that the validity of data analyses has improved compared to the original state of analysis. Still, despite the explicit guidelines, only 29.5% of all threats to validity were addressed properly. While the maturation and the optimal sequence threats are properly addressed in 35.8% and 38.8% of all studies in our sample respectively, the carryover threat is only modeled in about 3% of the observed cases. The lack of adherence to the analysis guidelines threatens the validity of the conclusions drawn from crossover design experiments.

Thu 24 Oct

Displayed time zone: Brussels, Copenhagen, Madrid, Paris change

11:00 - 12:30
11:00
20m
Full-paper
ChatGPT application in Systematic Literature Reviews in Software Engineering: an evaluation of its accuracy to support the selection activity
ESEM Technical Papers
Katia Romero Felizardo UTFPR-CP, Marcia Sampaio Lima Universidade do Estado do Amazonas - UEA, Anderson Deizepe UTFPR-CP, Tayana Conte Universidade Federal do Amazonas, Igor Steinmacher Northern Arizona University
11:20
20m
Full-paper
Is generalisation hindering the adoption of your findings?
ESEM Technical Papers
Rogardt Heldal Western Norway University of Applied Science
11:40
20m
Full-paper
Threats to Validity in Software Engineering -- hypocritical paper section or essential analysis?
ESEM Technical Papers
Patricia Lago Vrije Universiteit Amsterdam, Per Runeson Lund University, Qunying Song Lund University, Roberto Verdecchia University of Florence
Pre-print
12:00
15m
Vision and Emerging Results
Data extraction for systematic mapping study using a large language model - a proof-of-concept study in software engineering
ESEM Emerging Results, Vision and Reflection Papers Track
Katia Romero Felizardo UTFPR-CP, Igor Steinmacher Northern Arizona University, Marcia Sampaio Lima Universidade do Estado do Amazonas - UEA, Anderson Deizepe UTFPR-CP, Tayana Conte Universidade Federal do Amazonas, Monalessa P. Barcellos Federal University of EspĂ­rito Santo
12:15
15m
Vision and Emerging Results
Crossover Designs in Software Engineering Experiments: Review of the State of Analysis
ESEM Emerging Results, Vision and Reflection Papers Track
Julian Frattini Blekinge Institute of Technology, Davide Fucci Blekinge Institute of Technology, Sira Vegas Universidad Politecnica de Madrid
Link to publication DOI Pre-print Media Attached