Industry Practitioners’ Perspectives on AI Model Quality: Perceptions, Challenges, and Solutions
Artificial Intelligence (AI) is now ubiquitous. From self-driving vehicles navigating dense city streets to recommendation engines curating personalized content, AI powers applications across nearly every industry. With this broad adoption, assuring the quality of AI models, i.e., the machine-learning components that power these applications, is essential for building reliable and trustworthy systems. Historically, correctness has been the primary focus in AI model development. Yet industry AI models may require many other critical quality attributes. To understand industry perceptions, challenges, and solutions regarding different quality attributes of AI models, we identify nine key quality attributes and conduct interviews with fifteen AI industry practitioners from various countries, companies, and roles. Through the interviews, we summarize practitioners’ perceptions on the importance of quality attributes; for example, efficiency is priorized over correctness for real-time AI applications, and scalability and deployability are no longer AI developers’ primary concerns. Among the challenges we identify, data imbalance stands out as one of the major obstacles to maintaining model correctness and robustness & security; accordingly, our interviews reveal mitigation strategies such as active learning for data collection. We also validate our key findings via a survey of 50 AI industry practitioners, and most findings are well-acknowledged except one being marginally acknowledged. Perceptions of each quality attribute can guide researchers to focus on attributes valued by practitioners when designing new techniques, while avoiding approaches that improve one attribute at the expense of others deemed more critical.
Wed 10 JunDisplayed time zone: London change
15:30 - 17:00 | AI Systems Engineering 2Industry Papers / Research Papers at JMS 745 Chair(s): Jingyue Li Norwegian University of Science and Technology (NTNU) | ||
15:30 15mTalk | DeepParse: Hybrid Log Parsing with LLM-Synthesized Regex Masks Research Papers Pre-print | ||
15:45 15mTalk | MoEKD: Mixture-of-Experts Knowledge Distillation for Robust and High-Performing Compressed Code ModelsBest Paper Award Research Papers Md. Abdul Awal University of Saskatchewan, Mrigank Rochan University of Saskatchewan, Chanchal K. Roy University of Saskatchewan Pre-print | ||
16:00 15mTalk | HKI-RAG:Hierarchical Knowledge Indexing for Retrieval-Augmented Generation in Distributed Heterogeneous Architectures Research Papers Chenglin Zhang School of Artificial Intelligence, China University ofGeosciences (Beijing), Teng Long School of Artificial Intelligence, China University of Geosciences (Beijing) | ||
16:15 15mTalk | PLMGH: What Matters in PLM-GNN Hybrids for Code Classification and Vulnerability Detection Research Papers Taoufik Kaouthar El Idrissi Polytechnique Montreal, Edward Zulkoski Quantstamp, Mohammad Hamdaqa Polytechnique Montreal | ||
16:30 10mTalk | Engineering a Governance-Aware AI Sandbox: Design, Implementation, and Lessons Learned Industry Papers Muhammad Waseem Faculty of Information Technology and Communication Sciences, Tampere University, 33014 Tampere, Finland, Md Aidul Islam Faculty of Information Technology and CommunicationSciences, Tampere University, 33014 Tampere, Finland, Md Nasir Uddin Shuvo Faculty of Information Technology and CommunicationSciences, Tampere University, 33014 Tampere, Finland, Md Mahade Hasan Tampere University, Kai-Kristian Kemell Tampere University, Jussi Rasku Tampere University, Mika Saari Tampere University, Vilma Saari DIMECC Oy., Tampere, Finland, Roope Pajasmaa DIMECC Oy., Tampere, Finland, Markku Oivo DIMECC Oy., Tampere, Finland, Pekka Abrahamsson Tampere University | ||
16:40 15mTalk | Industry Practitioners’ Perspectives on AI Model Quality: Perceptions, Challenges, and Solutions Research Papers Chenyu Wang Singapore Management University, Zhou Yang University of Alberta; CIFAR AI Chair; Alberta Machine Intelligence Institute , Yunbo Lyu Singapore Management University, Ze Shi (Zane) Li University of Oklahoma, Dana Damian University of Victoria, David Lo Singapore Management University Pre-print | ||