Inference in Empirical Software Engineering

In this talk I’d like to explore what we mean by inference, specifically scientific inference, and how this is deployed in empirical software engineering. By inference I mean the process of reaching a conclusion on the basis of evidence including from observations and reasoning. Two points to note are (i) the conclusion goes beyond the evidence; it is an extension or generalization and (ii) the inferencing process is uncertain, thus the conclusion could be incorrect.

Within empirical software engineering research, the typical form of inference is statistical — usually, but not necessarily, some form of null hypothesis significance testing — within a deductive framework based upon a priori hypotheses. In contrast one could also proceed in an inductive fashion from particular empirical observations in order to reach more general conclusions. Machine learning is the most obvious example. Case studies and Grounded Theory are qualitative research examples of induction. The use of empirically-based inference has been accelerated by the widespread acceptance of evidence-based software engineering.

However, abduction is an alternative form of inference that might also be relevant to software engineering. Unlike induction, abduction places a strong emphasis on explanatory considerations. Therefore, I suggest that some openness to alternative and supplementary forms of inferencing could be valuable and potentially widen the reach of our research.

To flesh out these ideas, I explore a number of examples including: (in a semi-humorous vein) the citation impact of colons in titles, comparing different machine learning approaches for software defect prediction and the anchoring bias for software professionals making productivity estimates.

I conclude by making some practical suggestions. First for software engineering researchers we should: - be explicit about (i) how we are making inferences, (ii) the target domain, and (iii) the uncertainty associated with the inference - be open to alternative and supplementary forms of inference - in addition to prediction, value explanation more highly

And from the point of view of consumers of such research, I suggest we should: - determine the extent to which the inference might be viewed as predictive or explanatory (causal) - consider the certainty associated with any inference and the extent to which it is merited - try to assess the mappings between our problem domain and that specified/implied by the researcher - from the foregoing assess the risks and opportunities of relying upon a scientific inference