SSBSE 2022
Thu 17 - Fri 18 November 2022 Singapore
co-located with ESEC/FSE 2022

Methodology and Guidelines for Evaluating Multi-Objective Search-Based Software Engineering

Miqing Li, University of Birmingham, UK & Tao Chen, Loughborough University, UK

  

Abstract

Search-Based Software Engineering (SBSE) has been becoming an increasingly important research paradigm for automating and solving different software engineering tasks. When the considered tasks have more than one objective/criterion to be optimised, they are called multi-objective ones. In such a scenario, the outcome is typically a set of incomparable solutions (i.e., being Pareto non-dominated to each other), and then a common question faced by many SBSE practitioners is: how to evaluate the obtained sets by using the right methods and indicators in the SBSE context? In this tutorial, we seek to provide a systematic methodology and guideline for answering this question. We start off by discussing why we need formal evaluation methods/indicators for multi-objective optimisation problems in general, and the result of a survey on how they have been dominantly used in SBSE. This is then followed by a detailed introduction of representative evaluation methods and quality indicators used in SBSE, including their behaviors and preferences. In the meantime, we demonstrate the patterns and examples of potentially misleading usages/choices of evaluation methods and quality indicators from the SBSE community, highlighting their consequences. Afterwards, we present a systematic methodology that can guide the selection and use of evaluation methods and quality indicators for a given SBSE problem in general, together with pointers that we hope to spark dialogues about some future directions on this important research topic for SBSE. Lastly, we showcase several real-world multi-objective SBSE case studies, in which we demonstrate the consequences of incorrect use of evaluation methods/indicators and exemplify the implementation of the guidance provided.

Biographies

Dr Miqing Li is an assistant professor at the University of Birmingham and a Turing Fellow of the Alan Turing Institute, UK. Miqing’s research revolves around multi-objective optimisation. In general, his research consists of two parts: 1) basic research, namely, developing effective evolutionary algorithms for general challenging optimisation problems such as those with many objectives, complex constraints, numerous local/global optima, and expensive to evaluate, and 2) applied research, namely, designing customised search algorithms for practical problems in other fields such as those in software engineering, high-performance computing, neural architecture search, disassembly automation, emergency supply distribution, supply chain.

Dr. Tao Chen is currently a Lecturer (Assistant Professor) in Computer Science at the Department of Computer Science, Loughborough University, United Kingdom. He has broad research interests in software engineering, including but not limited to search-based software engineering (particularly the general aspects of SBSE), performance engineering, self-adaptive software systems, data-driven software engineering, and computational intelligence. Over the past decade, he has been working on specialising artificial/computational intelligence algorithms for understanding, improving, and evaluating the designs for engineering software systems together with their runtime behaviors. As the lead author, his work has been published in major Software Engineering journals and conferences, such as TSE, TOSEM, ICSE, FSE, and ASE.

Dates
Tracks
You're viewing the program in a time zone which is different from your device's time zone - change time zone

Thu 17 Nov

Displayed time zone: Beijing, Chongqing, Hong Kong, Urumqi change

09:00 - 10:30
Plenary + Keynote 1Keynotes at ERC SR 9
Chair(s): Mike Papadakis University of Luxembourg, Luxembourg
09:00
90m
Keynote
Applications of Search-based Software Testing to Trustworthy Artificial Intelligence
Keynotes
Lionel Briand University of Luxembourg; University of Ottawa
11:00 - 12:30
Session 1Research Papers / RENE / NIER at ERC SR 9
Chair(s): Ezekiel Soremekun SnT, University of Luxembourg
11:00
30m
Talk
Guess What: Test Case Generation for Javascript with Unsupervised Probabilistic Type Inference
Research Papers
Dimitri Stallenberg Delft University of Technology, Mitchell Olsthoorn Delft University of Technology, Annibale Panichella Delft University of Technology
Pre-print
11:30
30m
Talk
Improving Search-based Android Test Generation using Surrogate Models
Research Papers
Michael Auer University of Passau, Felix Adler University of Passau, Gordon Fraser University of Passau
12:00
30m
Talk
Applying Combinatorial Testing to Verification-Based Fairness Testing
RENE / NIER
Takashi Kitamura , Zhenjiang Zhao Graduate School of Informatics and Engineering, University of Electro-Communications, Tokyo, Japan, Takahisa Toda The University of Electro-Communications
14:00 - 15:30
Session 2Research Papers / Challenge Track at ERC SR 9
Chair(s): Renzo Degiovanni SnT, University of Luxembourg
14:00
30m
Talk
An Empirical Comparison of EvoSuite and DSpot for Improving Developer-Written Test Suites with Respect to Mutation Score
Research Papers
Muhammad Firhard Roslan University of Sheffield, José Miguel Rojas The University of Sheffield, Phil McMinn University of Sheffield
14:30
30m
Talk
Efficient Fairness Testing through Hash-Based Sampling
Research Papers
Zhenjiang Zhao Graduate School of Informatics and Engineering, University of Electro-Communications, Tokyo, Japan, Takahisa Toda The University of Electro-Communications, Takashi Kitamura
15:00
30m
Talk
Multi-Objective Genetic Improvement: A Case Study with EvoSuite
Challenge Track
James Callan UCL, Justyna Petke University College London
16:00 - 17:30
Session 3Research Papers at ERC SR 9
Chair(s): Mitchell Olsthoorn Delft University of Technology
16:00
30m
Talk
EvoAttack: An Evolutionary Search-based Adversarial Attack for Object Detection Models
Research Papers
Kenneth Chan Michigan State University, Betty H.C. Cheng Michigan State University
16:30
30m
Talk
Search-based Test Suite Generation for Rust
Research Papers
Vsevolod Tymofyeyev University of Passau, Gordon Fraser University of Passau

Fri 18 Nov

Displayed time zone: Beijing, Chongqing, Hong Kong, Urumqi change

11:00 - 12:30
Future of SSBSE 1Future of SBSE at Virtual 3 (Whova)
Chair(s): Thiago Ferreira University of Michigan - Flint
11:00
30m
Talk
ML is the new SBSE
Future of SBSE
Myra Cohen Iowa State University
11:30
30m
Talk
Reverse engineering the new SBSE
Future of SBSE
Tim Menzies North Carolina State University
14:00 - 15:30
TutorialTutorial at Virtual 3 (Whova)
Chair(s): Jeongju Sohn University of Luxembourg, Luxembourg
14:30
60m
Tutorial
Methodology and Guidelines for Evaluating Multi-Objective Search-Based Software Engineering
Tutorial
Miqing Li University of Birmingham, Tao Chen Loughborough University
Link to publication Pre-print
16:00 - 17:30
Keynote 2Keynotes at Virtual 3 (Whova)
Chair(s): Annibale Panichella Delft University of Technology
16:00
90m
Keynote
Genetic Improvement of Software
Keynotes
Justyna Petke University College London
18:30 - 20:00
Future of SSBSE 2Future of SBSE at Virtual 3 (Whova)
Chair(s): Giovani Guizzo University College London
18:30
30m
Talk
Online software safety: a new paradigm for SBSE research
Future of SBSE
Mark Harman Meta Platforms, Inc. and UCL
19:00
30m
Talk
"SSBSE 2050: 14-18 November, Oxia Palus, Mars"
Future of SBSE
Andrea Arcuri Kristiania University College and Oslo Metropolitan University
19:30
30m
Talk
Data Mining Algorithms Using/Used-by Optimisers: a DUO Approach to Software Engineering
Future of SBSE
Leandro Minku University of Birmingham, UK