Generative Reasoning vs. Conventional Analysis: How Well Do LLMs Detect Data Races in Concurrent Programs?
Concurrent programs are prone to data races due to nondeterministic thread interleavings, yet traditional detection tools still suffer from inherent trade-offs among precision, coverage, and applicability. Can Large Language Models (LLMs) offer a viable alternative? While LLMs have demonstrated strong code understanding capabilities, their effectiveness in concurrent program analysis remains largely unexplored. We present a comprehensive empirical study comparing LLM-based approaches with traditional static and dynamic analysis tools for data race detection. Our evaluation spans over 3,950 concurrent program files (1.4M+ LoC) from three complementary sources: standard benchmarks, Linux-kernel verification tasks, and large-scale real-world projects. We systematically evaluate multiple LLMs under direct prompting, Chain-of-Thought (CoT) reasoning, and a multi-stage agent workflow for race detection. We further investigate the impact of model families and parameter scales (4B–235B), and compare the optimal configuration against state-of-the-art concurrency analysis tools (e.g., TSan, OpenRace, and SVF). Results show that, under our evaluation setting, the LLMbased agent achieves the best F1 score with significantly higher precision and comparable recall, while requiring neither compilation nor execution. We also analyze cost-efficiency trade-offs and reveal that LLM hallucinations and traditional tool false alarms exhibit fundamentally different code patterns, indicating that combining generative reasoning with conventional analysis is a promising direction for advancing concurrent program analysis.
Sat 18 JulDisplayed time zone: Brisbane change
14:00 - 15:30 | Session 4: Software TestingResearch Track at Promenade Chair(s): Xinguo Feng The University of Queensland | ||
14:00 15mTalk | DFuzz: Differential Fuzzing for Silent Errors in LLM Operator Optimizations Research Track Jiayi Wang Nanjing University, Zihan Tang Nanjing University, Daohan Qu Nanjing University, Zenan Li ETH Zurich, Yuan Yao Nanjing University, Taolue Chen Birkbeck, University of London, Xiaoxing Ma Nanjing University | ||
14:15 15mTalk | Understanding Bugs in Vector Database Management Systems Research Track Yinglin Xie Huazhong University of Science and Technology, Xinyi Hou Huazhong University of Science and Technology, Yanjie Zhao Huazhong University of Science and Technology, Shenao Wang Huazhong University of Science and Technology, Kai Chen Huazhong University of Science and Technology, Haoyu Wang Huazhong University of Science and Technology | ||
14:30 15mTalk | Segmented Search for Maximum Error Detection in Floating-Point Arithmetic Expressions Research Track dmy Information Engineering University, Fei Li Information Engineering University, Jinchen Xu Information Engineering University, Hongru Yang Hunan University, Changsha, China, Tao Zhang Information Engineering University, Jiaxin Feng Information Engineering University, Bei Zhou Information Engineering University | ||
14:45 15mTalk | GapFuzz: Cross-Plane Divergence Fuzzing for Distributed SDN Controllers Research Track Moustapha Awwalou DIOUF SnT, University of Luxembourg, Samuel Ouya Cheikh Hamidou KANE Digital University, Jacques Klein University of Luxembourg, Tegawendé F. Bissyandé University of Luxembourg Pre-print | ||
15:00 15mTalk | HardRace: A Data Race Monitor for Production Use Research Track Xudong Sun , Zhuo Chen , Jingyang Shi Nanjing University, Yiyu Zhang Nanjing University, Peng Di Kunlunxin & UNSW Sydney, Fengwei Zhang Southern University of Science and Technology, Jianhua Zhao Nanjing University, China, Zhiqiang Zuo Nanjing University | ||
15:15 15mTalk | Generative Reasoning vs. Conventional Analysis: How Well Do LLMs Detect Data Races in Concurrent Programs? Research Track Xinyin Liao Xidian University, Zhiwei Lin Xidian University, Cheng Wen Xidian University, Jie Su Xidian University, Shengchao Qin Xidian University, Xiaoxue Ma City University of Hong Kong, Xiaofeng Li Beijing Institute of Control Engineering, Cong Tian Xidian University Media Attached | ||