Methodology and Guidelines for Evaluating Multi-Objective Search-Based Software Engineering

Miqing Li, University of Birmingham, UK & Tao Chen, Loughborough University, UK

Abstract

Search-Based Software Engineering (SBSE) has been becoming an increasingly important research paradigm for automating and solving different software engineering tasks. When the considered tasks have more than one objective/criterion to be optimised, they are called multi-objective ones. In such a scenario, the outcome is typically a set of incomparable solutions (i.e., being Pareto non-dominated to each other), and then a common question faced by many SBSE practitioners is: how to evaluate the obtained sets by using the right methods and indicators in the SBSE context? In this tutorial, we seek to provide a systematic methodology and guideline for answering this question. We start off by discussing why we need formal evaluation methods/indicators for multi-objective optimisation problems in general, and the result of a survey on how they have been dominantly used in SBSE. This is then followed by a detailed introduction of representative evaluation methods and quality indicators used in SBSE, including their behaviors and preferences. In the meantime, we demonstrate the patterns and examples of potentially misleading usages/choices of evaluation methods and quality indicators from the SBSE community, highlighting their consequences. Afterwards, we present a systematic methodology that can guide the selection and use of evaluation methods and quality indicators for a given SBSE problem in general, together with pointers that we hope to spark dialogues about some future directions on this important research topic for SBSE. Lastly, we showcase several real-world multi-objective SBSE case studies, in which we demonstrate the consequences of incorrect use of evaluation methods/indicators and exemplify the implementation of the guidance provided.

Biographies

Dr Miqing Li is an assistant professor at the University of Birmingham and a Turing Fellow of the Alan Turing Institute, UK. Miqing’s research revolves around multi-objective optimisation. In general, his research consists of two parts: 1) basic research, namely, developing effective evolutionary algorithms for general challenging optimisation problems such as those with many objectives, complex constraints, numerous local/global optima, and expensive to evaluate, and 2) applied research, namely, designing customised search algorithms for practical problems in other fields such as those in software engineering, high-performance computing, neural architecture search, disassembly automation, emergency supply distribution, supply chain.

Dr. Tao Chen is currently a Lecturer (Assistant Professor) in Computer Science at the Department of Computer Science, Loughborough University, United Kingdom. He has broad research interests in software engineering, including but not limited to search-based software engineering (particularly the general aspects of SBSE), performance engineering, self-adaptive software systems, data-driven software engineering, and computational intelligence. Over the past decade, he has been working on specialising artificial/computational intelligence algorithms for understanding, improving, and evaluating the designs for engineering software systems together with their runtime behaviors. As the lead author, his work has been published in major Software Engineering journals and conferences, such as TSE, TOSEM, ICSE, FSE, and ASE.