ICSME 2025
Sun 7 - Fri 12 September 2025 Auckland, New Zealand
Fri 12 Sep 2025 11:10 - 11:25 at Case Room 2 260-057 - Session 14 - Human Factors 2 Chair(s): Valeria Pontillo

Given the inherent non-deterministic nature of machine learning (ML) systems, their behavior in production environments can lead to unforeseen and potentially dangerous outcomes. For a timely detection of unwanted behavior and to prevent organizations from financial and reputational damage, monitoring these systems is essential. This paper explores the strategies, challenges, and improvement opportunities for monitoring ML systems from the practitioners’ perspective. We conducted a global survey of 91 ML practitioners to collect diverse insights into current ML monitoring practices. We aim to complement existing research through our qualitative and quantitative analyses, focusing on prevalent runtime issues, industrial monitoring and mitigation practices, key challenges, and desired enhancements in future monitoring tools. Our findings reveal that practitioners frequently struggle with runtime issues related to declining prediction quality, exceeding latency, and security violations. While most prefer automated monitoring for the increased efficiency, many still rely on manual approaches due to the complexity or the lack of appropriate automation solutions. Practitioners report that the initial setup and configuration of monitoring tools is often complicated and challenging, particularly when integrating with ML systems and setting alert thresholds. Moreover, practitioners find that monitoring adds extra workload, strains resources, and causes alert fatigue. The desired improvements from the practitioners’ perspective are: improved support for performance and fairness monitoring, recommendations for resolving runtime issues, and automated generation and deployment of monitors. These insights offer valuable guidance for the future development of ML monitoring tools that are better aligned with practitioners’ needs.

Fri 12 Sep

Displayed time zone: Auckland, Wellington change

10:30 - 12:00
10:30
15m
Software Fairness Testing in Practice
Research Papers Track
10:45
15m
Refactoring Deep Learning Code: A Study of Practices and Unsatisfied Tool Needs
Research Papers Track
Siqi Wang Zhejiang University, Xing Hu Zhejiang University, Bei Wang Zhejiang University, China, Wenxin Yao Zhejiang University, Xin Xia Zhejiang University, Xinyu Wang Zhejiang University
11:00
10m
CodeWatcher: IDE Telemetry Data Extraction Tool for Understanding Coding Interactions with LLMs
Tool Demonstration Track
Manaal Ramadan Basha The University of British Columbia, Aimee M. Ribeiro Federal University of Para, Jeena Javahar The University of British Columbia, Gema Rodriguez-Perez The University of British Columbia, Cleidson de Souza Federal University of Pará, Brazil
11:10
15m
Understanding Practitioners’ Perspectives on Monitoring Machine Learning Systems
Industry Track
Hira Naveed Monash University, John Grundy Monash University, Chetan Arora Monash University, Hourieh Khalajzadeh Deakin University, Australia, Omar Haggag Monash University, Australia
Pre-print
11:25
15m
Using AI-based Coding Assistants in Practice: State of Affairs, Perceptions, and Ways Forward
Journal First Track
Agnia Sergeyuk JetBrains Research, Yaroslav Golubev JetBrains Research, Timofey Bryksin JetBrains Research, Iftekhar Ahmed University of California at Irvine
Link to publication DOI Pre-print
11:40
10m
An Empirical Study of GenAI Adoption in Open-Source Game Development: Tools, Tasks, and Developer Challenges
Registered Reports
Xiang Chen University of Waterloo, Wenhan Zhu Huawei Canada, Guoshuai Shi University of Waterloo, Michael W. Godfrey University of Waterloo, Canada
Pre-print
11:50
10m
Evaluating the Comprehension of the Stackage ecosystem: A Comparison Between VR and 2D Visualizations
Registered Reports
David Moreno-Lumbreras Universidad Rey Juan Carlos, Paul Leger Universidad Católica del Norte, Chile, Sergio Montes-León Universidad Rey Juan Carlos, Jesus M. Gonzalez-Barahona Universidad Rey Juan Carlos, Gregorio Robles Universidad Rey Juan Carlos