A Unified and Holistic Classification Scheme for Software Engineering Research
Background/Context: Classifications in meta-research enable researchers to cope with an increasing body of scientific knowledge. They provide a framework for, e.g., distinguishing methods, reports, reproducibility, and evaluation in a knowledge field as well as a common terminology. Likewise, they are important for the development of a research field. Both eases sharing, understanding and evolution of knowledge. In software engineering (SE), there are several classifications that describe the nature of SE research (e.g., classifications of research methods, replication types, types of SE contributions). Regarding the classification and consolidation of the huge body of knowledge in SE research, a generally applicable classification scheme is crucial. However, it is unclear whether existing classification schemes are suitable candidates. Moreover, the commonalities and differences among different classification schemes have rarely been studied. Due to the fact that classifications are documented textual, it is hard to catalog, reuse, and compare them. To the best of our knowledge, there is no research work so far that addresses documentation and systematic investigation of classifications in SE meta-research.
Objective/Aim: In this research work, we investigate provenance of SE research classifications and construct a unified, generally applicable classification scheme for SE meta-research by collecting and documenting a set of existing classification schemes and unifying their classes and categories. To validate the generality (i.e., whether it is both – general and specific enough) and appropriateness (i.e., whether it fully and correctly covers all relevant aspects of collected classifications), we compare our unified classification scheme with a set of collected classification employing existing metrics. Likewise, we evaluate the applicability of our scheme in a user study.
Method: Our research process is divided into three phases: construction, validation, and evaluation phase. For the construction phase, we perform a systematic literature review to identify, collect, and analyze a set of established SE research classifications. In the validation phase, we analyze individual categories and classes of selected and included papers. We use quantitative metrics from literature to conduct and assess the unification process to build a generally applicable classification scheme for SE research. Lastly, we investigate the applicability of the unified classification scheme in the evaluation phase. Therefore, we perform a workshop session followed by user studies w.r.t. investigations about reliability, correctness of classification tasks according to the unified scheme and user satisfaction (i.e., ease of use). The findings form the basis for further empirical studies that address the applicability of the unified scheme to real-world systems.