Tutorial: How (not) to Analyze Software Engineering Experiments: From Anti-Patterns to Solutions
Experimentation is a key aspect of science and engineering, yet it remains one of the major stumbling blocks in software engineering. Although many experiments are conducted today, ensuring their quality-whether they involve human subjects or not-remains a persistent concern for the trustworthiness of results. Researchers have raised concerns about the correct use of statistical methods for many years, these issues often persist due to two main factors: the inherent complexity of empirical studies in our field, and the unique characteristics of software engineering, which lead to some experimentation issues being conceived differently than in other disciplines. This tutorial focuses on the analysis of experimental data, helping participants avoid common pitfalls and anti-patterns while improving the quality and reliability of their results. It is not intended as a data analysis course, but rather reviews key issues identified in published software engineering experiments, providing guidance based on over 25 years of experience running experiments.
