ICST 2025 Keynote Speaker

Repurposing automated testing for ML software development and deployment

Speaker: Atif Memon

Abstract: ML-powered features in software are notoriously challenging to test. As the behaviors of an ML feature emerge, so do user-level bugs. Traditional testing methods conducted during code development, or metrics collected during model development fail to provide an accurate view of such bugs. Moreover, online metrics collected during A/B experiments do not directly expose them. I will talk about how we are filling this gap by repurposing automated testing for ML software development and deployment. We leverage large volumes of system test cases, automatically obtained with their test oracles, to derive defect classes, which represent a partition of the input space on which the ML-based software does measurably worse. These defect classes are reported early during development, thereby enabling rapid improvement.

Atif Memon

Biography: Atif Memon is a Distinguished Engineer at Apple, where he leads the development of machine learning-driven tools for evaluating conventional and AI-powered software, processing millions of evaluations daily. His work includes developing methodologies for automatically authoring and maintaining tests for ML systems, and adapting automated testing to meet the challenges of machine learning software development. He has developed a holistic approach for evaluating emergent behavior in ML-based information retrieval systems. Additionally, he has created algorithms for flakiness scoring that substantially reduce test flakiness with minimal loss in fault detection. Before joining Apple, Dr. Memon was a Professor in the Department of Computer Science at the University of Maryland, where he founded the Event Driven Software Lab (EDSL) and published over 100 research papers on software testing and engineering. He designed GUITAR, a model-based GUI testing tool, and served on numerous program and steering committees. Dr. Memon earned his Ph.D. from the University of Pittsburgh and has held visiting researcher positions at Google, the Chinese Academy of Sciences, and Tata Research. Dr. Memon’s contributions have been recognized with the NSF CAREER Award, among other honors. He has received grants from the NSF, NIH, NASA, and DARPA for his research work.