Navigating the landscape of AI test methods using taxonomy-based selection
Due to broad deployment of AI systems in risk-prone domains and AI regulations coming into effect, systematic risk- and quality assessments of AI systems have become increasingly important. Conducting such assessments involves identifying relevant quality criteria for a given AI system and selecting test methods, i.e., procedures for collecting and evaluating evidences and measurable quantities that fit the identified criteria. This selection process can be challenging due to the high complexity of the test method landscape and, in the context of independent audits, due to potential conflicts of interests between the involved stakeholders. To address this challenge, we expand existing frameworks on AI assessments with a systematic, taxonomy-based method of selecting test methods in independent audits suited for the given AI system and its application context. We evaluate our taxonomy on a subset of OECD metrics catalogue and demonstrate its applicability with two use cases.
Tue 29 AprDisplayed time zone: Eastern Time (US & Canada) change
11:00 - 12:30 | |||
11:00 15mTalk | Using Drift Planning to Improve Safety of Visual Navigation in Unmanned Aerial Vehicles RAIE Jeffrey Hansen Carnegie Mellon Software Engineering Institute, Sebastian Echeverria Carnegie Mellon Software Engineering Institute, Lena Pons Carnegie Mellon Software Engineering Institute, Lihan Zhan Carnegie Mellon Software Engineering Institute, Gabriel A. Moreno Carnegie Mellon University Software Engineering Institute, P: Grace Lewis Carnegie Mellon Software Engineering Institute | ||
11:15 15mTalk | LLM-AQuA-DiVeR: LLM-Assisted Quality Assurance Through Dialogues on Verifiable Specification with Requirement Owners RAIE P: Shohei Mitani Georgetown University, Salonee Moona Triple Point Security, Shinichiro Matsuo Georgetown University, Eric Burger Virginia Tech | ||
11:30 12mTalk | Towards Ensuring Responsible AI for Medical Device Certification RAIE P: Giulio Mallardi University of Bari, Luigi Quaranta University of Bari, Italy, Fabio Calefato University of Bari, Filippo Lanubile University of Bari | ||
11:42 12mTalk | Navigating the landscape of AI test methods using taxonomy-based selection RAIE Maximilian Pintz Fraunhofer Institute for Intelligent Analysis and Information Systems, University of Bonn, Anna Schmitz Fraunhofer Institute for Intelligent Analysis and Information Systems, Rebekka Görge Fraunhofer Institute for Intelligent Analysis and Information Systems, P: Sebastian Schmidt Fraunhofer Institute for Intelligent Analysis and Information Systems, Daniel Becker , Maram Akila Fraunhofer Institute for Intelligent Analysis and Information Systems, Lamarr Institute, Michael Mock Fraunhofer Institute for Intelligent Analysis and Information Systems | ||
11:54 12mTalk | Responsible AI in the Software Industry: A Practitioner-Centered Perspective RAIE P: Matheus de Morais Leça University of Calgary, Mariana Pinheiro Bento University of Calgary, Ronnie de Souza Santos University of Calgary Pre-print | ||
12:06 12mTalk | The Privacy Pillar - A Conceptual Framework for Foundation Model-based Systems RAIE Tingting Bi The University of Melbourne, Guangsheng Yu University of Technology Sydney, Qin Wang CSIRO Data61 |