The goal of this tutorial is to provide an overview of challenges and approaches for V&V in microservices systems. Specifically, we present dependability assessment approaches that employ operational data obtained from production-level application performance management (APM) tools, giving access to operational workload profiles, architectural information, failure models, and security intrusions. We use this data to automatically create and configure dependability assessments based on models, load tests, and resilience benchmarks. The focus of this tutorial is on approaches that employ production usage, because these approaches provide more accurate recommendations for microservice architecture dependability assessment than approaches that do not consider production usage.

We present an overview of (1) the challenges and opportunities for dependability assessment and software performance anti-patterns based improvement in microservices, (2) selected approaches based on operational data to assess dependability. The dependability focus of this tutorial is on performance, scalability, resilience, survivability, and security. Particularly, we present an interactive demo of an automated approach for the evaluation of a domain-based scalability and security metric assessment that is based on the microservice architecture ability to satisfy the performance requirement under load and/or intrusions as implemented by the PPTAM tool (https://github.com/pptam). We illustrate the approach by presenting experimental results using a benchmark microservice architecture. We also present the application of a multivariate approach for detection of scalability choke points and the application of software performance antipatterns to provide recommendations for performance improvement.

Track records/bios of organizer and additional instructor:

Alberto Avritzer received a Ph.D. in Computer Science from the University of California, Los Angeles, an M.Sc. in Computer Science for the Federal University of Minas Gerais, Brazil, and the B.Sc. in Computer Engineering from the Technion, Israel Institute of Technology. He is the founder and CEO of Esulabsolutions, Inc., which specializes in software scalability, security and survivability assessment of large industrial systems. He served as Lead Performance Engineer at Sonatype. He held a Senior Member of the Technical Staff in the Software Engineering Department position at Siemens Corporate Research, Princeton, New Jersey for 11 years, where he published extensively on monitoring and management of mission-critical systems for survivability. Before moving to Siemens Corporate Research, he spent 13 years at AT&T Bell Laboratories, where he developed tools and techniques for performance testing and analysis. He spent the summer of 1987 at IBM Research, at Yorktown Heights. His research interests are in software engineering, particularly software testing, monitoring and rejuvenation of smoothly degrading systems, and metrics to assess software architecture, and he has published over 70 papers in journals and refereed conference proceedings in those areas (http://dblp.dagstuhl.de/pers/hd/a/Avritzer:Alberto). He is a Senior Member of ACM.

Barbara Russo is full professor at the Free University of Bozen-Bolzano, Italy. She holds a PhD in Mathematics of the University of Trento, Italy. She was visiting researcher at the Max-Plank Institute für Mathematik in Bonn, Germany and research fellow at the University of Liverpool, UK. She is member of ISERN the International Research NEtwork in Software Engineering. She has a good record of conferences’ organization and chairing in the Software Engineering field. In 2019, she was co-chair the 13th International Symposium on Empirical Software Engineering and Measurement (ESEM2019), the Doctoral Symposium of the 27th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), and the workshop GAUSS at ISSRE2019,ì. She has co-organized tutorials on performance engineering at ICSA, ICPE, ISSRE. She co-authored more than 120 papers and is reviewer for top journals and conferences in the field. She has extensively collaborated with companies in research projects attracting more than 900,000 Euro as principal investigator. Her research interest is in AI for software engineering, system behavior and performance and system testing and reliability (http://www.inf.unibz.it/~russo/ )

Matteo Camilli is an assistant professor at the Faculty of Computer Science of the Free University of Bozen-Bolzano (Italy). His current research activity focuses on Formal Methods and Software Engineering. He is especially interested in: formal verification; model-based testing techniques; uncertainty quantification; integrated methods and tools to improve dependability of distributed, service-based, autonomous, cyber-physical, and ML-enabled systems. He publishes papers in international journals and in proceedings of international conferences. He serves as member of the program committee of international conferences. He regularly organizes the International Workshop on Formal Approaches for Advanced Computing Systems (FAACS) co-located with the European Conference on Software Architecture (ECSA) and the technical track on Software Architecture (SA-TTA) of the SIGAPP/ACM Symposium on Applied Computing (SAC). He his part of the organizing committee of the IEEE International Conference on Software Architecture (ICSA 2022, ICSA 2023) and the Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS 2023).

Andrea Janes is a researcher at the Faculty of Computer Science of the Free University of Bozen/Bolzano. He holds a Master’s degree in Business Informatics from the University of Vienna (Austria). He obtained his PhD cum laude in Computer Science at the University of Klagenfurt (Austria) with a thesis on the measurement of software processes in a non-invasive way. His research interests are Agile and Lean software development, non-invasive techniques for measuring software development processes, software testing and data analysis. Since 2017, he has been a member of the “Smart Data Factory”, a technology transfer project at the NOI Techpark. He is author and co-author of over 50 chapters of books, articles and other contributions.

André van Hoorn is a senior researcher with the Department of Informatics at the University of Hamburg. Before, André was with the Institute of Software Technology at the University of Stuttgart, Germany, where he was an interim professor for reliable software systems from 2015 to 2017. He received his Ph.D. degree (with distinction) from Kiel University, Germany (2014) and his Master’s degree (Dipl.- Inform.) from the University of Oldenburg, Germany (2007). André’s research focuses on novel methods, techniques, and tools for designing, operating, and evolving trustworthy distributed software systems. Of particular interest are quality attributes such as performance, reliability, and resilience–and how they can be assessed and optimized using a smart combination of model-based and measurement-based approaches. Currently, André investigates challenges and opportunities to apply such approaches in the context of continuous software engineering and DevOps. Andre has published over 50 papers in journals and refereed conferences in those areas (http://dblp.dagstuhl.de/pers/hd/h/Hoorn:Andr=eacute=_van). He is the principal investigator of several research projects and is actively involved in community activities, e.g., in the scope of the Research Group of the Standard Performance Evaluation Corporation (SPEC).

Catia Trubiani is Assistant Professor at the Gran Sasso Science Institute (GSSI), L’Aquila, Italy. Previously she worked at the University of Rome "Tor Vergata" within the framework of a European FP7 project. Before joining the GSSI, she has been with various international research institutes like the Imperial College of London (UK) and the Karlsruhe Institute of Technology (Germany). Among various national and international research projects in which she has been involved, she is principal investigator of the GSSI unit for the MIUR-PRIN project SEDUCE (Young Line action) and for the MIUR-FISR project MVM-Adapt. She received the Exceptional Reviewer Award based on Review Quality Collector tool for ICSA (International Conference on Software Architecture, ranked A in the core classification) in 2021, the Best Research Paper Award at ICPE (International Conference on Performance Engineering) in 2011, and recognized also as 10-year Most Influential Paper Award in ICPE 2021. Her main research interests include the quantitative modelling and analysis of interacting heterogeneous distributed systems. She is especially interested in model-based performance analysis and feedback on software architectures, model-based refactoring by means of software performance antipatterns, and trade-off quality analysis and optimization of software systems. More information is available here: http://cs.gssi.it/catia.trubiani

Outline with approximate timings:

• Part 1 (30m minutes): challenges of V&V in microservices systems, with particular focus on performance and scalability assessment
• Part 2 (30 minutes): quantitative assessment of microservice architecture deployment alternatives by automated performance testing: the PPTAM approach
• Part 3 (30 minutes): selected approaches for continuous dependability assessment using operational profiles and survivability modeling. We introduce the application of a multivariate approach for detection of scalability choke points and the application of software performance antipatterns to provide recommendations for performance improvement.
• Part 4 (90 minutes): demo
  • write performance tests with Locust.io
  • run scalability assessment experiments with PPTAM
  • intepret the results of PPTAM

Target audience: The tutorial is self-contained and accessible to researchers and practitioners in the area of software architecture and performance engineering. No specific background is required, in addition to basic Software Architecture background. Takeaways: Tutorial attendees will learn how to exploit operational data for V&V of microservices systems with special focus on performance and scalability assessment. History of the tutorial: The current tutorial format is new and will be presented, if approved, for the first time at ECSA 2022. Related/shorter versions of the tutorial, keynotes, presentations, and related papers were presented in several venues, among them: ICPE 2013, ISSRE 2014, ICPE 2017, ITC 2017, ISSRE 2017, ICPE 2018, ECSA 2018, ISSRE 2018, ISSRE 2019, ICPE 2020, ICSA 2020, ISSRE 2020, ICSA 2022.

References:

• A. Avritzer, V. Ferme, A. Janes, B. Russo, H. Schulz, A. van Hoorn, A Quantitative Approach for the Assessment of Microservice Architecture Deployment Alternatives by Automated Performance Testing. ECSA 2018: 159-174
• A. Avritzer, D. S. Menasché, V. Rufino, B. Russo, A. Janes, V. Ferme, A. van Hoorn, H. Schulz, PPTAM: Production and Performance Testing Based Application Monitoring. ICPE Companion 2019: 39-40
• A. Avritzer, D. S. Menasché, V. Rufino, B. Russo, A. Janes, V. Ferme, A. van Hoorn, H. Schulz, Scalability Assessment of Microservice Architectural Configurations: A Domain-based Approach Leveraging Operational Profiles and Load Tests, Journal of Systems and Software, July 2020
• Alberto Avritzer, Ricardo Britto, Catia Trubiani, Barbara Russo, Andrea Janes, Matteo Camilli, Andre van Hoorn, Robert Heinrich, Martina Rapp, and Jorg Henß. A multivariate characterization and detection of software performance antipatterns. In Proceedings of the ACM/SPEC International Conference on Performance Engineering, ICPE ’21, page 61–72, New York, NY, USA, 2021a. ACM. ISBN 9781450381949. doi: 10.1145/3427921.3450246. URL https://doi.org/10.1145/3427921.3450246
• Alberto Avritzer, Matteo Camilli, Andrea Janes, Barbara Russo, Jasmin Jahic, Andre van Hoorn, Ricardo Britto, and Catia Trubiani. PPTAMλ: What, where, and how of cross-domain scalability assessment. In 2021 IEEE 18th International Conference on Software Architecture Companion (ICSA-C), pages 62–69, 2021b. doi: 10.1109/ICSA-C52384.2021.00016
• Matteo Camilli, Andrea Janes, and Barbara Russo. Automated test-based learning and verification of performance models for microservices systems. Journal of Systems and Software, page 111225, 2022b. ISSN 0164-1212. doi: https://doi.org/10.1016/j.jss.2022.111225.
• Alberto Avritzer, Ricardo Britto, Catia Trubiani, Matteo Camilli, Andrea Janes, Barbara Russo, Andre van Hoorn, Robert Heinrich, Martina Rapp, J ̈org Henß, and Ram Kishan Chalawadi. Scalability testing automation using multivariate characterization and detection of software performance antipatterns. Journal of Systems and Software, page 111446, 2022. ISSN 0164-1212. doi: https://doi.org/10.1016/j.jss.2022.111446.