Robert Feldt, Martin Shepperd

Abstract:

In the wider scientific community e.g., Medicine and Psychology, there is a long tradition of questioning the methods used to analyse empirical data. In statistics there has been a very lively debate with some “camps” arguing for abandonment of classical methods while others counter that current methods can suffice if used appropriately. With a few exceptions, there has been less awareness in the empirical software engineering (ESE) community. In this session we want to initiate such a discussion in order to (i) consider what we can do differently in order to modernize data analysis within ESE and (ii) understand what are current barriers to change.

Session Goals:

• Identify and characterise the different “camps” and schools of thoughts on data analysis for scientific knowledge
• highlight challenges to as well as enablers for changing analysis methods (AMs) in ESE
• Develop a manifesto for modernizing/changing AMs in ESE

Development of the Session: (How will the session be conducted? How much interaction?)

We will provide background and then present cases for further discussion:

1. Dichotomizing scientific knowledge (avoiding Null Hypothesis Significance Testing)
2. Isolated paper “islands” vs building chains of ESE evidence
3. Point estimates vs distributions

Discussion in small groups which is then summarized in the larger group/session Group prioritization and “prediction market” of challenges/enablers/mitigations

What means for interaction will be used or required?

Mainly group discussions but also 100-dollar prioritization/prediction market sessions to clarify what researchers see as main versus less central challenges and ways forward.

Background and recommended reading:

Recommended reading:

1. McShane, Blakeley B., et al. “Abandon statistical significance.” The American Statistician 73.sup1 (2019): 235-245.

For additional background:

1. Colquhoun, David. “An investigation of the false discovery rate and the misinterpretation of p-values.” Royal Society open science 1.3 (2014): 140216.
2. Berkson, Joseph. “Tests of significance considered as evidence.” Journal of the American Statistical Association 37.219 (1942): 325-335.
3. Hunter, John E., and Frank L. Schmidt. “Cumulative research knowledge and social policy formulation: The critical role of meta-analysis.” Psychology, Public Policy, and Law 2.2 (1996): 324.

Expected Outcomes and Plan for Continuing the Work beyond ISERN:

We aim to write a joint paper, with the most interested and contributing ISERN members of the session, that provides an analysis of current challenges in modernizing AMs in ESE but also presents a manifesto for change and proposes concrete ways forward.