Towards a Responsible AI Metrics Catalogue: A Collection of Metrics for AI AccountabilityDistinguished paper Award Candidate
Artificial Intelligence (AI), particularly through the advent of large-scale generative AI (GenAI) models such as Large Language Models (LLMs), has become a transformative element in contemporary technology. While these models have unlocked new possibilities, they simultaneously present significant challenges, such as concerns over data privacy and the propensity to generate misleading or fabricated content. Current frameworks for Responsible AI (RAI) often fall short in providing the granular guidance necessary for tangible application, especially for \textit{Accountability}—a principle that is pivotal for ensuring transparent and auditable decision-making, bolstering public trust, and meeting increasing regulatory expectations. This study bridges the \textit{Accountability gap} by introducing a comprehensive metrics catalogue, formulated through a systematic multivocal literature review (MLR) that integrates findings from both academic and grey literature. Our catalogue delineates process metrics that underpin procedural integrity, resource metrics that provide necessary tools and frameworks, and product metrics that reflect the outputs of AI systems. This tripartite framework is designed to operationalize Accountability in AI, with a special emphasis on addressing the intricacies of GenAI. The proposed metrics catalogue provides a robust framework for instilling Accountability in AI systems. It offers practical, actionable guidance for organizations, thereby shaping responsible practices in the field.
Sun 14 AprDisplayed time zone: Lisbon change
16:00 - 18:00 | Generative AI EngineeringIndustry Talks / Research and Experience Papers at Pequeno Auditório Chair(s): Ipek Ozkaya Carnegie Mellon University | ||
16:00 15mTalk | Developer Experiences with a Contextualized AI Coding Assistant: Usability, Expectations, and Outcomes Research and Experience Papers Gustavo Pinto Federal University of Pará (UFPA) and Zup Innovation, Cleidson de Souza Federal University of Pará, Brazil, Thayssa Rocha Zup Innovation & UFPA, Igor Steinmacher Northern Arizona University, Alberto de Souza Zup Innovation, Edward Monteiro StackSpot | ||
16:15 10mTalk | Privacy and Copyright Protection in Generative AI: A Lifecycle Perspective Research and Experience Papers Dawen (David) Zhang CSIRO's Data61, Boming Xia CSIRO's Data61 & University of New South Wales, Yue Liu CSIRO's Data61 & University of New South Wales, Xiwei (Sherry) Xu Data61, CSIRO, Thong Hoang CSIRO's Data61, Zhenchang Xing CSIRO's Data61, Mark Staples CSIRO, Australia, Qinghua Lu Data61, CSIRO, Liming Zhu CSIRO’s Data61 | ||
16:25 10mIndustry talk | Innovating Translation: Lessons Learned from BWX Generative Language Engine Industry Talks | ||
16:35 15mTalk | Towards a Responsible AI Metrics Catalogue: A Collection of Metrics for AI AccountabilityDistinguished paper Award Candidate Research and Experience Papers Boming Xia CSIRO's Data61 & University of New South Wales, Qinghua Lu Data61, CSIRO, Liming Zhu CSIRO’s Data61, Sung Une (Sunny) Lee CSIRO's Data61, Yue Liu CSIRO's Data61 & University of New South Wales, Zhenchang Xing CSIRO's Data61 Pre-print | ||
16:50 10mLive Q&A | GenAI : Q&A Research and Experience Papers | ||
17:00 60mPanel | Industry Panel Industry Talks |