Machine Learning Systems Engineering: Retrospective of Five-Year Activities in Japan

Dr. Hiroshi Maruyama

Abstract

The process of developing machine learning (ML)-based systems is in many aspects different from the process of developing conventional software systems, where the accumulated knowledge on Software Engineering (SE) can guide. In order to facilitate the discussions on what existing SE practices can be applied and what are missing for ML-based systems, we launched a community on Machine Learning Systems Engineering in 2017. Our activities cover a wide range of aspects regarding developing and operating ML-based systems, from requirement development, testing and quality assurance, tools and computing infrastructure, to operating and management issues. This talk reflects on the five year activities, reviews the original goals and what we have achieved and what we have not, and discuss the future directions.

Short Biography

Dr. Hiroshi Maruyama has spent 26 years in IBM Research, Tokyo Research Laboratory, working on various computer science areas such as artificial intelligence, natural language processing, machine translation, hand-writing recognition, multimedia, XML, Web Services, and security. He was the director of IBM Tokyo Research Laboratory from 2006 to 2009. From 2011 to 2016, he was a professor at the Institute of Statistical Mathematics where he worked on projects related to big data, statistics, and their impacts on society. He joined Preferred Networks, Inc. in April 2016 as the chief strategy officer. His current research interests include practical applications of machine learning, social implications of information technology and machine learning, and computer science and statistics in general. Currently he is an Executive Fellow at Kao Corporation, a PFN Fellow at Preferred Networks, and a project professor at the Research into Artifacts, Center for Engineering at the University of Tokyo.

Improving Development of ML-Enabled Systems through Software Architecture

Dr. Grace A. Lewis

Abstract

Developing software systems that contain machine learning (ML) components (ML-enabled systems, or ML systems for short) requires an end-to-end perspective that considers the unique life cycle of these components — from data acquisition, to model training, to model deployment and evolution. However, a problem is that ML system development typically split into three roles, with three different and often separate workflows: data scientists build models; software engineers integrate models into ML systems; and then operations staff deploy, operate, and monitor the ML systems. Because these roles operate separately and often speak different languages, there are opportunities for mismatch between the assumptions made by each role with respect to the elements of the ML-enabled system, and the actual guarantees provided by each element, which leads to system failure. While simply better collaboration between teams is a valid solution to this problem, using software architecture design as the set of activities and artifacts that promote and record collaborative decision making is a much stronger and sustainable solution.

In this talk I will first share the outcomes of two practitioner studies that highlight the problems related to treating ML systems development as a model-centric instead of a system-centric activity. I will then present a set of software architecture practices that can lead to successful ML systems. I will close the talk with some thoughts on remaining gaps in both research and practice that I hope will inspire current and future software engineering and software architecture research.

Short Biography

Grace Lewis is a Principal Researcher at the Carnegie Mellon Software Engineering Institute (SEI) where she conducts applied research on how software engineering and software architecture principles, practices and tools need to evolve in the face of emerging technologies.

She is the principal investigator for the Automating Mismatch Detection and Testing in Machine Learning Systems project that is developing toolsets to support these two activities, in addition to other projects that are advancing the state of the practice in software engineering for machine learning (SE4ML). Grace is also the lead for the Tactical and AI-Enabled Systems (TAS) applied research and development team at the SEI that is creating and transitioning innovative solutions, principles, and best practices for

• architecting and developing systems to support teams operating at the tactical edge in resource-constrained environments
• engineering AI software systems
• using AI/ML at the edge for improved capabilities and mission support

She is currently VP of the IEEE Computer Society Technical & Conference Activities Board (T&C), Diversity and Inclusion (D&I) Vice-Chair for the IEEE Computer Society Technical Community on Software Engineering (TCSE), Alternate Representative for IEEE-CS on the ABET CSAB Board of Directors, as well as an ABET Evaluator for Computer Science undergraduate programs. Grace holds a B.Sc. in Software Systems Engineering and a Post-Graduate Specialization in Business Administration from Icesi University in Cali, Colombia; a Master in Software Engineering from Carnegie Mellon University; and a Ph.D. in Computer Science from Vrije Universiteit Amsterdam.

On the Search for Effective Metamorphic Relations: Overview, Challenges and Opportunities

Prof. Shing-Chi Cheung

Abstract

A major challenge in testing software like artificial intelligent, data-centric, and service-oriented software is the test oracle problem, which occurs when the expected output of a given program input is hard to determine. Instead of examining individual test results, Metamorphic Testing leverages domain-specific relations, called Metamorphic Relations, to address the test oracle problem via a series of test executions. Metamorphic testing is reported to be one of the most popular testing techniques in many application domains such as artificial intelligence, bioinformatics, and search engines. A recent literature review finds that metamorphic testing has been most extensively utilized to test web services, computer graphics, and simulation, making up 40% of related publications. It is also reported to be the most popular testing technique for AI systems. Metamorphic testing has received wide interest from the community. Over 800 research articles on "metamorphic testing" published since 2021 are indexed by Google Scholar. In this talk, we review the key concepts in the deployment of metamorphic testing. We demonstrate the design of effective metamorphic relations for service-oriented software using an example of synchronization bug detection in decentralized applications (DApps) on blockchain. We discuss the major challenges and various research opportunities.

Short Biography

Shing-Chi Cheung received his doctoral degree in Computing from the Imperial College London. After that, he joined the Hong Kong University of Science and Technology (HKUST), where he is a professor of Computer Science and Engineering. He founded the CASTLE research group at HKUST and co-founded in 2006 the International Workshop on Automation of Software Testing (AST), which is now an annual IEEE international conference. He was the General Chair of the 22nd ACM SIGSOFT International Symposium on the Foundations of Software Engineering (FSE 2014), the General Co-chair of APSEC 2012, and the Program Co-chair of APSEC 1996 and 1997. He was an editorial board member of the IEEE Transactions on Software Engineering (TSE, 2006-9). His research interests focus on the testing and analysis for applications on mobile, web, deep learning, open-source repositories, and blockchains. He is an ACM distinguished member. More information about his CASTLE research group can be found at here.