Towards an OSF-based Registered Report Template for Software Engineering Controlled ExperimentsTechnical Paper
Context: The empirical software engineering (ESE) community has contributed to improving experimentation over the years. However, there is still a lack of rigor in describing controlled experiments, hindering reproducibility and transparency. Registered Reports (RR) have been discussed in the ESE community to address these issues. A RR registers a study’s hypotheses, methods, and/or analyses before execution, involving peer review and potential acceptance before data collection. This helps mitigate problematic practices such as p-hacking, publication bias, and inappropriate post hoc analysis. Objective: This paper presents initial results toward establishing an RR template for Software Engineering controlled experiments using the Open Science Framework (OSF). Method: We analyzed templates of selected OSF RR type in light of documentation guidelines for controlled experiments. Results: The observed lack of rigor motivated our investigation of OSF-based RR types. Our analysis showed that, although one of the RR types aligned with many of the documentation suggestions contained in the guidelines, none of them covered the guidelines comprehensively. The study also highlights limitations in OSF RR template customization. Conclusion: Despite progress in ESE, planning and documenting experiments still lack rigor, compromising reproducibility. Adopting OSF-based RRs is proposed. However, no currently available RR type fully satisfies the guidelines. Establishing RR-specific guidelines for SE is deemed essential.
Mon 13 AprDisplayed time zone: Brasilia, Distrito Federal, Brazil change
09:00 - 10:30 | Opening, Keynote, Controlled ExperimentsWSESE at Oceania III Chair(s): Silvia Abrahão Universitat Politècnica de València, Patricia Lago Vrije Universiteit Amsterdam, Sira Vegas Universidad Politecnica de Madrid | ||
09:00 15mDay opening | Welcome by the Organizers WSESE Patricia Lago Vrije Universiteit Amsterdam, Silvia Abrahão Universitat Politècnica de València, Sira Vegas Universidad Politecnica de Madrid | ||
09:15 50mKeynote | We don’t know how to best use LLMs for qualitative analysis… But maybe we know how not to WSESE Carolyn Seaman University of Maryland Baltimore County | ||
10:05 18mFull-paper | Towards an OSF-based Registered Report Template for Software Engineering Controlled ExperimentsTechnical Paper WSESE Ana B. M. Bett State University of Maringá, Thais S. Nepomuceno State University of Maringá, Edson OliveiraJr State University of Maringá, Maria Teresa Baldassarre Department of Computer Science, University of Bari , Valdemar Graciano Neto Federal University of Goiás, Marcos Kalinowski Pontifical Catholic University of Rio de Janeiro (PUC-Rio) Pre-print | ||
10:23 7mPanel | Discussion of Controlled Experiments WSESE | ||