On the Effectiveness of SBSE Techniques through Instance Space Analysis

Aldeida Aleti

Abstract - Search-Based Software Engineering is now a mature area with numerous techniques developed to tackle some of the most challenging software engineering problems, from requirements to design, testing, fault localisation, and automated program repair. SBSE techniques have shown promising results, giving us hope that one day it will be possible for the tedious and labour intensive parts of software development to be completely automated, or at least semi-automated. In this talk, I will focus on the problem of objective performance evaluation of SBSE techniques. To this end, I will introduce Instance Space Analysis (ISA), which is an approach to identify features of SBSE problems that explain why a particular instance is difficult for an SBSE technique. ISA can be used to examine the diversity and quality of the benchmark datasets used by most researchers, and analyse the strengths and weaknesses of existing SBSE techniques. The instance space is constructed to reveal areas of hard and easy problems, and enables the strengths and weaknesses of the different SBSE techniques to be identified. I will present on how ISA enabled us to identify the strengths and weaknesses of SBSE techniques in two areas: Search-Based Software Testing and Automated Program Repair. Finally, I will end my talk with future directions of the objective assessment of SBSE techniques.

Bio - Aldeida Aleti is an Associate Professor at the Faculty of Information Technology, Monash University in Australia, and the Associate Dean of Engagement and Impact. Aldeida’s research is in the area of search-based software engineering (SBSE), with a particular focus on what makes software engineering problems (design, testing, program repair) hard to optimise and designing approaches that make it easier to apply SBSE techniques to new problems. Aldeida has published more than 50 papers in top optimisation and software engineering venues, and has served as PC member and organising committee at both SE and optimisation conferences, such as ASE, ICSE, GECCO, FSE, SSBSE. Aldeida has attracted more than 2.5M in competitive funding to conduct research in the areas of fairness testing of ML-based healthcare systems and for developing search based methods for testing autonomous vehicles. Aldeida was awarded the prestigious Discovery Early Career Researcher (DECRA) Award from the Australian Research Council, and she has received multiple “best paper” and “best reviewer” awards.

How Empirical Research Supports Tool Development: A Retrospective Analysis and new Horizons

Massimiliano Di Penta

Shared with ESEM

Abstract - Empirical research provides two-fold support to the development of approaches and tools aimed at supporting software engineers. On the one hand, empirical studies help to understand a phenomenon or a context of interest. On the other hand, studies compare approaches and evaluate how software engineers benefit from them. Over the past decades, there has been a tangible evolution in how empirical evaluation is conducted in software engineering. This is due to multiple reasons. On the one hand, the research community has matured a lot thanks also to guidelines developed by several researchers. On the other hand, the large availability of data and artifacts, mainly from the open-source, has made it possible to conduct larger evaluations, and in some cases to reach study participants. In this keynote, I will first overview how empirical research has been used over the past decades to evaluate tools, and how this is changing over the years. I will also emphasize the importance of combining quantitative and qualitative evaluations, and how sometimes “depth” turns out to be more useful than just “breadth”. I will also emphasize how research is not a straightforward path, and negative results are often an essential component for future advances. Last, but not least, I will emphasize how the role of empirical evaluation is changing with the pervasiveness of artificial intelligence methods in software engineering research.

Bio - Massimiliano Di Penta is a full professor at the University of Sannio, Italy. His research interests include software maintenance and evolution, mining software repositories, empirical software engineering, search-based software engineering, and software testing. He is an author of over 300 papers that appeared in international journals, conferences, and workshops. He has received several awards for research and service, including four ACM SIGSOFT Distinguished paper awards. He serves and has served in the organizing and program committees of more than 100 conferences, including ICSE, FSE, ASE, and ICSME. Among others, he has been program co-chair of ASE 2017, ESEC/FSE 2021, and will be program co-chair of ICSE 2023. He is co-editor in chief of the Journal of Software: Evolution and Processes edited by Wiley, editorial board member of ACM Transactions on Software Engineering and Methodology and Empirical Software Engineering Journal edited by Springer, and has served the editorial board of the IEEE Transactions on Software Engineering.