CAIN 2024
Sun 14 - Mon 15 April 2024 Lisbon, Portugal
co-located with ICSE 2024
Sun 14 Apr 2024 16:35 - 16:50 at Pequeno Auditório - Generative AI Engineering Chair(s): Ipek Ozkaya

Artificial Intelligence (AI), particularly through the advent of large-scale generative AI (GenAI) models such as Large Language Models (LLMs), has become a transformative element in contemporary technology. While these models have unlocked new possibilities, they simultaneously present significant challenges, such as concerns over data privacy and the propensity to generate misleading or fabricated content. Current frameworks for Responsible AI (RAI) often fall short in providing the granular guidance necessary for tangible application, especially for \textit{Accountability}—a principle that is pivotal for ensuring transparent and auditable decision-making, bolstering public trust, and meeting increasing regulatory expectations. This study bridges the \textit{Accountability gap} by introducing a comprehensive metrics catalogue, formulated through a systematic multivocal literature review (MLR) that integrates findings from both academic and grey literature. Our catalogue delineates process metrics that underpin procedural integrity, resource metrics that provide necessary tools and frameworks, and product metrics that reflect the outputs of AI systems. This tripartite framework is designed to operationalize Accountability in AI, with a special emphasis on addressing the intricacies of GenAI. The proposed metrics catalogue provides a robust framework for instilling Accountability in AI systems. It offers practical, actionable guidance for organizations, thereby shaping responsible practices in the field.

Sun 14 Apr

Displayed time zone: Lisbon change

16:00 - 18:00
Generative AI EngineeringIndustry Talks / Research and Experience Papers at Pequeno Auditório
Chair(s): Ipek Ozkaya Carnegie Mellon University
16:00
15m
Talk
Developer Experiences with a Contextualized AI Coding Assistant: Usability, Expectations, and Outcomes
Research and Experience Papers
Gustavo Pinto Federal University of Pará (UFPA) and Zup Innovation, Cleidson de Souza Federal University of Pará, Brazil, Thayssa Rocha Zup Innovation & UFPA, Igor Steinmacher Northern Arizona University, Alberto de Souza Zup Innovation, Edward Monteiro StackSpot
16:15
10m
Talk
Privacy and Copyright Protection in Generative AI: A Lifecycle Perspective
Research and Experience Papers
Dawen (David) Zhang CSIRO's Data61, Boming Xia CSIRO's Data61 & University of New South Wales, Yue Liu CSIRO's Data61 & University of New South Wales, Xiwei (Sherry) Xu Data61, CSIRO, Thong Hoang CSIRO's Data61, Zhenchang Xing CSIRO's Data61, Mark Staples CSIRO, Australia, Qinghua Lu Data61, CSIRO, Liming Zhu CSIRO’s Data61
16:25
10m
Industry talk
Innovating Translation: Lessons Learned from BWX Generative Language Engine
Industry Talks
Vanilson Burégio Bureau Works, A: Iverson Pereira Bureau Works, Henrique Cabral
16:35
15m
Talk
Towards a Responsible AI Metrics Catalogue: A Collection of Metrics for AI AccountabilityDistinguished paper Award Candidate
Research and Experience Papers
Boming Xia CSIRO's Data61 & University of New South Wales, Qinghua Lu Data61, CSIRO, Liming Zhu CSIRO’s Data61, Sung Une (Sunny) Lee CSIRO's Data61, Yue Liu CSIRO's Data61 & University of New South Wales, Zhenchang Xing CSIRO's Data61
Pre-print
16:50
10m
Live Q&A
GenAI : Q&A
Research and Experience Papers

17:00
60m
Panel
Industry Panel
Industry Talks