Modeling Resilience of Collaborative AI Systems
A Collaborative Artificial Intelligence System (CAIS) performs actions in collaboration with the human to achieve a common goal. CAISs can use a trained AI model to control human-system interaction, or they can use human interaction to dynamically learn from humans in an online fashion. In online learning with human feedback, the AI model evolves by monitoring human interaction through the system sensors in the learning state, and actuates the autonomous components of the CAIS based on the learning in the operational state. Therefore, any disruptive event affecting these sensors may affect the AI model’s ability to make accurate decisions and degrade the CAIS performance. Consequently, it is of paramount importance for CAIS managers to be able to automatically track the system performance to understand the resilience of the CAIS against such disruptive events. In this paper, we provide a new framework to model CAIS performance when the system experiences a disruptive event. With our framework, we introduce a model of performance evolution of CAIS. The model is equipped with a set of measures that aim to support CAIS managers in the decision process to achieve the required resilience of the system. We tested our framework on a real-world case study of a robot collaborating online with the human, when the system is experiencing a disruptive event. The case study shows that our framework can be adopted in CAIS and integrated into the online execution of the CAIS activities.
Mon 15 AprDisplayed time zone: Lisbon change
16:00 - 18:00 | System QualitiesResearch and Experience Papers / Industry Talks at Pequeno Auditório Chair(s): Andrei Paleyes Department of Computer Science and Technology, Univesity of Cambridge | ||
16:00 10mTalk | Modeling Resilience of Collaborative AI Systems Research and Experience Papers Diaeddin Rimawi Free University of Bozen-Bolzano, Antonio Liotta Free University of Bozen-Bolzano, Marco Todescato Fraunhofer Italia, Barbara Russo | ||
16:10 10mTalk | Seven Failure Points When Engineering a Retrieval Augmented Generation System Research and Experience Papers Scott Barnett Applied Artificial Intelligence Institute, Deakin University, Stefanus Kurniawan Deakin University, Srikanth Thudumu Deakin University, Zach Brannelly Deakin University, Mohamed Abdelrazek Deakin University, Australia | ||
16:20 15mTalk | POLARIS: A framework to guide the development of Trustworthy AI systems Research and Experience Papers Maria Teresa Baldassarre Department of Computer Science, University of Bari , Domenico Gigante SER&Practices and University of Bari, Marcos Kalinowski Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Azzurra Ragone University of Bari | ||
16:35 15mTalk | Worst-Case Convergence Time of ML Algorithms via Extreme Value Theory Research and Experience Papers A: Saeid Tizpaz-Niari University of Texas at El Paso, A: Sriram Sankaranarayanan University of Colorado, Boulder | ||
16:50 15mTalk | Is Your Anomaly Detector Ready for Change? Adapting AIOps Solutions to the Real World Research and Experience Papers Lorena Poenaru-Olaru TU Delft, Natalia Karpova TU Delft, Luís Cruz Delft University of Technology, Jan S. Rellermeyer Leibniz University Hannover, Arie van Deursen Delft University of Technology | ||
17:05 15mTalk | Novel Contract-based Runtime Explainability Framework for End-to-End Ensemble Machine Learning Serving Research and Experience Papers Minh-Tri Nguyen Aalto University, Hong-Linh Truong Aalto University, Tram Truong-Huu Singapore Institute of Technology | ||
17:20 10mIndustry talk | Trustworthy AI: Industry-Guided Tooling of the Methods Industry Talks Zakaria Chihani CEA, LIST, France | ||
17:30 15mLive Q&A | System Qualities: Q&A Session Research and Experience Papers | ||
17:45 15mDay closing | Closing Research and Experience Papers Jan Bosch Chalmers University of Technology | ||