ICST 2025 will host two keynote speeches delivered by distinguished experts Atif Memon and Antonia Bertolino. Key details about each keynote are outlined below.
Keynote 1: Repurposing automated testing for ML software development and deployment
Speaker: Atif Memon
Abstract: ML-powered features in software are notoriously challenging to test. As the behaviors of an ML feature emerge, so do user-level bugs. Traditional testing methods conducted during code development, or metrics collected during model development fail to provide an accurate view of such bugs. Moreover, online metrics collected during A/B experiments do not directly expose them. I will talk about how we are filling this gap by repurposing automated testing for ML software development and deployment. We leverage large volumes of system test cases, automatically obtained with their test oracles, to derive defect classes, which represent a partition of the input space on which the ML-based software does measurably worse. These defect classes are reported early during development, thereby enabling rapid improvement.
Atif Memon
Biography: Atif Memon is a Distinguished Engineer at Apple, where he leads the development of machine learning-driven tools for evaluating conventional and AI-powered software, processing millions of evaluations daily. His work includes developing methodologies for automatically authoring and maintaining tests for ML systems, and adapting automated testing to meet the challenges of machine learning software development. He has developed a holistic approach for evaluating emergent behavior in ML-based information retrieval systems. Additionally, he has created algorithms for flakiness scoring that substantially reduce test flakiness with minimal loss in fault detection. Before joining Apple, Dr. Memon was a Professor in the Department of Computer Science at the University of Maryland, where he founded the Event Driven Software Lab (EDSL) and published over 100 research papers on software testing and engineering. He designed GUITAR, a model-based GUI testing tool, and served on numerous program and steering committees. Dr. Memon earned his Ph.D. from the University of Pittsburgh and has held visiting researcher positions at Google, the Chinese Academy of Sciences, and Tata Research. Dr. Memon’s contributions have been recognized with the NSF CAREER Award, among other honors. He has received grants from the NSF, NIH, NASA, and DARPA for his research work.
Keynote 2: LLMs: killers or boosters of software testing research?
Speaker: Antonia Bertolino
Abstract: After the big waves in the last two decades of leveraging AI and ML-based technologies for pushing test automation, in the latest couple of years the new big thing is undoubtedly LLM-assisted testing. LLMs are increasingly used for test case generation at differing levels of the testing process, for oracle derivation, for improving test code, as well as for debugging and code repairing. Several studies already show that LLMs can be an effective collaborator in different testing activities, but also warn about possible latent issues. Beyond the specific technique or tool, in this keynote I will talk on how LLMs are already largely transforming testing research. Questions that I would like to put on the discussion table are: can LLMs really substitute human testers? And to what extent can they solve the longstanding challenge of test automation? on what problems should software testing research focus? The talk will be based on a critical reflection of white and grey literature.
Antonia Bertolino
