DFuzz: Differential Fuzzing for Silent Errors in LLM Operator Optimizations
Modern large language model (LLM) training frameworks widely employ operator optimizations to improve efficiency. These optimizations are typically assumed to be numerically equivalent to their standard counterparts. However, due to the inherent imprecision of floating-point arithmetic, even mathematically equivalent implementations can introduce small per-step numerical deviations that silently accumulate during training. Such deviations constitute silent errors: they trigger no crashes or exceptions, yet cause optimized and unoptimized training runs to diverge over time. This paper presents \textsc{DFuzz}, a differential fuzzing framework for systematically detecting and quantifying silent errors in operator optimizations. \textsc{DFuzz} searches for worst-case numerical deviations by exploring both the training configuration space and the input space. It employs a two-stage pipeline: \emph{configuration fuzzing}, which uses a genetic algorithm to identify training settings that amplify deviations, followed by \emph{input fuzzing}, which further maximizes deviation via projected gradient-based search. We evaluate \textsc{DFuzz} on 44 operator optimizations across three major LLM training frameworks. The results show that \textsc{DFuzz} reveals silent errors in 16 operators (12 previously undocumented), and that both fuzzing stages significantly outperform random baselines by up to $13.33\times$. \textsc{DFuzz} uncovers three classes of silent errors: floating-point precision accumulation, undocumented algorithmic approximations, and implementation defects in LLM-generated code.
Sat 18 JulDisplayed time zone: Brisbane change
14:00 - 15:30 | Session 4: Software TestingResearch Track at Promenade Chair(s): Xinguo Feng The University of Queensland | ||
14:00 15mTalk | DFuzz: Differential Fuzzing for Silent Errors in LLM Operator Optimizations Research Track Jiayi Wang Nanjing University, Zihan Tang Nanjing University, Daohan Qu Nanjing University, Zenan Li ETH Zurich, Yuan Yao Nanjing University, Taolue Chen Birkbeck, University of London, Xiaoxing Ma Nanjing University | ||
14:15 15mTalk | Understanding Bugs in Vector Database Management Systems Research Track Yinglin Xie Huazhong University of Science and Technology, Xinyi Hou Huazhong University of Science and Technology, Yanjie Zhao Huazhong University of Science and Technology, Shenao Wang Huazhong University of Science and Technology, Kai Chen Huazhong University of Science and Technology, Haoyu Wang Huazhong University of Science and Technology | ||
14:30 15mTalk | Segmented Search for Maximum Error Detection in Floating-Point Arithmetic Expressions Research Track dmy Information Engineering University, Fei Li Information Engineering University, Jinchen Xu Information Engineering University, Hongru Yang Hunan University, Changsha, China, Tao Zhang Information Engineering University, Jiaxin Feng Information Engineering University, Bei Zhou Information Engineering University | ||
14:45 15mTalk | GapFuzz: Cross-Plane Divergence Fuzzing for Distributed SDN Controllers Research Track Moustapha Awwalou DIOUF SnT, University of Luxembourg, Samuel Ouya Cheikh Hamidou KANE Digital University, Jacques Klein University of Luxembourg, Tegawendé F. Bissyandé University of Luxembourg Pre-print | ||
15:00 15mTalk | HardRace: A Data Race Monitor for Production Use Research Track Xudong Sun , Zhuo Chen , Jingyang Shi Nanjing University, Yiyu Zhang Nanjing University, Peng Di Kunlunxin & UNSW Sydney, Fengwei Zhang Southern University of Science and Technology, Jianhua Zhao Nanjing University, China, Zhiqiang Zuo Nanjing University | ||
15:15 15mTalk | Generative Reasoning vs. Conventional Analysis: How Well Do LLMs Detect Data Races in Concurrent Programs? Research Track Xinyin Liao Xidian University, Zhiwei Lin Xidian University, Cheng Wen Xidian University, Jie Su Xidian University, Shengchao Qin Xidian University, Xiaoxue Ma City University of Hong Kong, Xiaofeng Li Beijing Institute of Control Engineering, Cong Tian Xidian University Media Attached | ||