This program is tentative and subject to change.
LLM-based mutation testing is rapidly emerging as a promising testing technology, but existing approaches typically rely on a fixed set of mutations as few-shot examples or none at all. This can result in generic low-quality mutations, missed context-specific mutation patterns, substantial numbers of redundant and uncompilable mutants, and limited semantic similarity to real bugs. To overcome these limitations, we introduce SMART. SMART integrates retrieval-augmented generation (RAG) on a vectorized dataset of real-world bugs, focused code chunking, and supervised fine-tuning using mutations coupled with real-world bugs. We conducted an extensive empirical study of SMART using 1,991 real-world Java bugs from the Defects4J and ConDefects datasets, comparing SMART to the state-of-the-art LLM-based approaches, LLMut and LLMorpheus.
The results reveal that SMART substantially improves mutation validity, effectiveness, and efficiency (even enabling small-scale 7B-scale models to match/surpass large models like GPT-4o). We also demonstrate that SMART significantly impacts downstream Software Engineering applications to test case prioritization and fault localization. More specifically, SMART improves validity (weighted average generation rate) from 42.89% to 65.6%. It raises the non-duplicate rate from 87.38% to 95.62%, and the compilable rate from 88.85% to 90.21%. In terms of effectiveness, it achieves a real bug detection rate of 92.61% (vs. 57.86% for LLMut) and improves the average Ochiai coefficient from 25.61% to 38.44%. For fault localization, it locates 64 more bugs by MUSE and 57 bugs by Metallaxis as Top-1.
This program is tentative and subject to change.
Wed 8 JulDisplayed time zone: Eastern Time (US & Canada) change
14:00 - 15:30 | |||
14:00 20mTalk | Red Teaming LLMs via Linguistic-Aware Fuzzing Research Papers Shuai Yuan University of Electronic Science and Technology of China, Nian Luo University Of Electronic Science And Technology Of China, Jingling Sun University of Electronic Science and Technology of China, Yihao Huang National University of Singapore, Singapore, Chengyu Zhang Loughborough University | ||
14:20 10mTalk | MIMIC-Py: An Extensible Tool for Personality-Driven Automated Game Testing with Large Language Models Tool Demonstrations | ||
14:30 10mTalk | Towards Automated Test Adaptation in Fork Ecosystems via Large Language Models Ideas, Visions and Reflections Mukelabai Mukelabai Ruhr University Bochum, Keanu-Wesley Schurkus Ruhr University Bochum, Yannic Noller Ruhr University Bochum, Thorsten Berger Ruhr University Bochum | ||
14:40 20mTalk | Boosting LLMs for Mutation Generation Research Papers Bo Wang Beijing Jiaotong University, Ming Deng Beijing Jiaotong University, Mingda Chen Beijing Jiaotong University, Chengran Yang Singapore Management University, Singapore, Youfang Lin Beijing Jiaotong University, Mark Harman Meta Platforms, Inc. and UCL, Mike Papadakis University of Luxembourg, Jie M. Zhang Mistral AI and King's College London | ||
15:00 20mTalk | LLM-Assisted Input-Requirement-Aware Differential Testing of Array Programming Frameworks Research Papers Zhichao Zhou School of Information Science and Technology, ShanghaiTech University, Jingzhu He ShanghaiTech University | ||
15:20 10mTalk | AISysRev - LLM-based Tool for Title-abstract Screening Tool Demonstrations Aleksi Huotala University of Helsinki, Miikka Kuutila LUT University, Olli-Pekka Turtio University of Helsinki, Simo Sipilä University of Helsinki, Mika Mäntylä University of Helsinki | ||