Per Runeson, Dag Sjøberg, Viktoria Stray

Title: Threats to Validity in Software Engineering – hypocritical paper section or essential analysis?

Co-Chairs: Per Runeson, Lund University, Dag Sjøberg and Viktoria Stray, University of Oslo

Abstract:

Threats to validity (TTV) is by many scholars considered a mandatory section in empirical software engineering papers. However, the purpose and recommended practices for such a section differ widely. Feldt and Magazinius made an initial survey in 2010 and Sjøberg and Bergersen recently reported about the state of practice for construct validity in five high-quality journals. Storey et al. adapted a framework to software engineering, trading different types of validity for different types of studies.

We aim to bring this discussion to the ISERN community, to explore the needs for improved practice and guidance to support such improvements. Specifically:

• What is the overall purpose of the TTV section?
• Can the TTV analysis increasingly become a pro-active part of the study design, rather than an afterthought?
• Can the different types of threats be traded against each other, depending on the goal and type of study?
• Can the TTV analysis be more standardized, without being superficial?

Session Goals:

To discuss current practices empirical software engineering literature with respect to threat to validity sections. The aim is to impact future conference and journal reviews, most specifically ESEM’24.

Development of the Session: (How will the session be conducted? How much interaction?)

Session attendants are expected to be familiar with basic TTV concepts. Hence, only a short introduction to the topic is given. Then group discussion will be held, addressing the above questions. Finally, these are reported back to the full session.

Background and recommended reading:

R. Feldt and A. Magazinius. Validity threats in empirical software engineering research - an initial survey. In Proceedings of the 22nd International Conference on Software Engineering & Knowledge Engineering (SEKE’2010), Redwood City, San Francisco Bay, CA, USA, July 1 - July 3, 2010, pages 374–379. Knowledge Systems Institute Graduate School, 2010.

Sjøberg, D.I.K., Bergersen, G.R., 2023. Construct validity in software engineering. IEEE Transactions on Software Engineering 49, 1374– 1396. doi:10.1109/tse.2022.3176725.

Storey, M.A., Ernst, N.A., Williams, C., Kalliamvakou, E., 2020. The who, what, how of software engineering research: a socio-technical framework. Empirical Software Engineering 25, 4097–4129. doi:10.1007/s10664-020-09858-z.

Expected Outcomes and Plan for Continuing the Work beyond ISERN: * Increased awareness of TTV issues, leading to improved empirical studies. * Collaboration plans on further literature work, along the lines of Sjøberg and Bergersen.