DevMuT: Testing Deep Learning Framework via Developer Expertise-Based Mutation
Deep learning (DL) frameworks are the fundamental infrastructure for various DL applications. Framework defects can profoundly cause disaster accidents, thus requiring sufficient detection. In previous studies, researchers adopt DL models as test inputs combined with mutation to generate more diverse models. Though these studies demonstrate promising results, most detected defects are considered trivial (i.e., either treated as edge cases or ignored by the developers). To identify important bugs that matter to developers, we propose a novel DL framework testing method DevMuT, which generates models adopting mutation operators and constraints derived from developers’ expertise. DevMuT simulates common operations of developers in development and detects more diverse defects within more stages of the DL model lifecycle (e.g., model training and inference). We evaluate the performance of DevMuT on three widely used DL frameworks (i.e., PyTorch, JAX, and MindSpore) with 29 DL models from nine types of industry tasks. The experiment results show that DevMuT outperforms state-of-the-art baselines: it can achieve at least 71.68% average improvement on the diversity of generated models and 28.20% average improvement on the legal rates of generated models. Moreover, DevMuT detects 117 defects, 63 of which are confirmed, 24 are fixed, and eight are of high value confirmed by developers. Finally, DevMuT has been deployed in the MindSpore community since December 2023. These demonstrate the effectiveness of DevMuT in detecting defects that are close to the real scenes and concerned by developers.
Thu 31 OctDisplayed time zone: Pacific Time (US & Canada) change
15:30 - 16:30 | SE for AI 3Research Papers at Magnoila Chair(s): Nafiz Imtiaz Khan Department of Computer Science, University of California, Davis | ||
15:30 15mTalk | DevMuT: Testing Deep Learning Framework via Developer Expertise-Based Mutation Research Papers Yanzhou Mu , Juan Zhai University of Massachusetts at Amherst, Chunrong Fang Nanjing University, Xiang Chen Nantong University, Zhixiang Cao Xi'an Jiaotong University, Peiran Yang State Key Laboratory for Novel Software Technology, Nanjing University, China, Yinglong Zou Nanjing University, Tao Zheng Nanjing University, Zhenyu Chen Nanjing University | ||
15:45 15mTalk | Mutation-Based Deep Learning Framework Testing Method in JavaScript Environment Research Papers Yinglong Zou Nanjing University, Juan Zhai University of Massachusetts at Amherst, Chunrong Fang Nanjing University, Jiawei Liu University of Illinois at Urbana-Champaign, Tao Zheng Nanjing University, Zhenyu Chen Nanjing University | ||
16:00 15mTalk | DynaMO: Protecting Mobile DL Models through Coupling Obfuscated DL Operators Research Papers Mingyi Zhou Monash University, Xiang Gao Beihang University, Xiao Chen University of Newcastle, Chunyang Chen TU Munich, John Grundy Monash University, Li Li Beihang University | ||
16:15 15mTalk | GlitchProber: Advancing Effective Detection and Mitigation of Glitch Tokens in Large Language Models Research Papers Zhibo Zhang Huazhong University of Science and Technology, Wuxia Bai Huazhong University of Science and Technology, Yuxi Li Huazhong University of Science and Technology, Mark Huasong Meng National University of Singapore, Kailong Wang Huazhong University of Science and Technology, Ling Shi Nanyang Technological University, Li Li Beihang University, Jun Wang Post Luxembourg, Haoyu Wang Huazhong University of Science and Technology |