ASE 2024
Sun 27 October - Fri 1 November 2024 Sacramento, California, United States
Thu 31 Oct 2024 15:30 - 15:45 at Magnoila - SE for AI 3 Chair(s): Nafiz Imtiaz Khan

Deep learning (DL) frameworks are the fundamental infrastructure for various DL applications. Framework defects can profoundly cause disaster accidents, thus requiring sufficient detection. In previous studies, researchers adopt DL models as test inputs combined with mutation to generate more diverse models. Though these studies demonstrate promising results, most detected defects are considered trivial (i.e., either treated as edge cases or ignored by the developers). To identify important bugs that matter to developers, we propose a novel DL framework testing method DevMuT, which generates models adopting mutation operators and constraints derived from developers’ expertise. DevMuT simulates common operations of developers in development and detects more diverse defects within more stages of the DL model lifecycle (e.g., model training and inference). We evaluate the performance of DevMuT on three widely used DL frameworks (i.e., PyTorch, JAX, and MindSpore) with 29 DL models from nine types of industry tasks. The experiment results show that DevMuT outperforms state-of-the-art baselines: it can achieve at least 71.68% average improvement on the diversity of generated models and 28.20% average improvement on the legal rates of generated models. Moreover, DevMuT detects 117 defects, 63 of which are confirmed, 24 are fixed, and eight are of high value confirmed by developers. Finally, DevMuT has been deployed in the MindSpore community since December 2023. These demonstrate the effectiveness of DevMuT in detecting defects that are close to the real scenes and concerned by developers.

Thu 31 Oct

Displayed time zone: Pacific Time (US & Canada) change

15:30 - 16:30
SE for AI 3Research Papers at Magnoila
Chair(s): Nafiz Imtiaz Khan Department of Computer Science, University of California, Davis
15:30
15m
Talk
DevMuT: Testing Deep Learning Framework via Developer Expertise-Based Mutation
Research Papers
Yanzhou Mu , Juan Zhai University of Massachusetts at Amherst, Chunrong Fang Nanjing University, Xiang Chen Nantong University, Zhixiang Cao Xi'an Jiaotong University, Peiran Yang State Key Laboratory for Novel Software Technology, Nanjing University, China, Yinglong Zou Nanjing University, Tao Zheng Nanjing University, Zhenyu Chen Nanjing University
15:45
15m
Talk
Mutation-Based Deep Learning Framework Testing Method in JavaScript Environment
Research Papers
Yinglong Zou Nanjing University, Juan Zhai University of Massachusetts at Amherst, Chunrong Fang Nanjing University, Jiawei Liu University of Illinois at Urbana-Champaign, Tao Zheng Nanjing University, Zhenyu Chen Nanjing University
16:00
15m
Talk
DynaMO: Protecting Mobile DL Models through Coupling Obfuscated DL Operators
Research Papers
Mingyi Zhou Monash University, Xiang Gao Beihang University, Xiao Chen University of Newcastle, Chunyang Chen TU Munich, John Grundy Monash University, Li Li Beihang University
16:15
15m
Talk
GlitchProber: Advancing Effective Detection and Mitigation of Glitch Tokens in Large Language Models
Research Papers
Zhibo Zhang Huazhong University of Science and Technology, Wuxia Bai Huazhong University of Science and Technology, Yuxi Li Huazhong University of Science and Technology, Mark Huasong Meng National University of Singapore, Kailong Wang Huazhong University of Science and Technology, Ling Shi Nanyang Technological University, Li Li Beihang University, Jun Wang Post Luxembourg, Haoyu Wang Huazhong University of Science and Technology