Defects4C: Benchmarking Large Language Model Repair Capability with C/C++ Bugs
Automated Program Repair (APR) plays a critical role in enhancing the quality and reliability of software systems. While substantial progress has been made in Java-based APR, largely facilitated by benchmarks like Defects4J, there remains a significant gap in research on C/C++ program repair, despite the widespread use of C/C++ and the prevalence of associated vulnerabilities. This gap is primarily due to the lack of high-quality, open-source benchmarks tailored for C/C++.
To address this issue, we introduce \textbf{\textit{Defects4C}}, a comprehensive and executable benchmark specifically designed for C/C++ program repair. Our dataset is constructed from real-world C/C++ repositories and includes a large collection of bug-relevant commits (\textbf{9M} in total), \textbf{248} high-quality buggy functions, and \textbf{102} vulnerable functions, all paired with test cases for reproduction. These resources enable rigorous evaluation of repair techniques and support the retraining of learning-based approaches for enhanced performance.
Using \textbf{\textit{Defects4C}}, we conduct a comprehensive empirical study evaluating the effectiveness of \textbf{24} state-of-the-art large language models (LLMs) in repairing C/C++ faults. Our findings offer valuable insights into the strengths and limitations of current LLM-based APR techniques in this domain, highlighting both the need for more robust methods and the critical role of Defects4C in advancing future research.
Mon 17 NovDisplayed time zone: Seoul change
11:00 - 12:30 | |||
11:00 10mTalk | Defects4C: Benchmarking Large Language Model Repair Capability with C/C++ Bugs Research Papers Jian Wang Nanyang Technological University, Xiaofei Xie Singapore Management University, Qiang Hu Tianjin University, Shangqing Liu Nanjing University, Jiongchi Yu Singapore Management University, Jiaolong Kong Singapore Management University, Yi Li Nanyang Technological University Pre-print | ||
11:10 10mTalk | MORepair: Teaching LLMs to Repair Code via Multi-Objective Fine-Tuning Journal-First Boyang Yang Yanshan University, Haoye Tian Aalto University, Jiadong Ren Yanshan University, Hongyu Zhang Chongqing University, Jacques Klein University of Luxembourg, Tegawendé F. Bissyandé University of Luxembourg, Claire Le Goues Carnegie Mellon University, Shunfu Jin Yanshan University Link to publication DOI Pre-print | ||
11:20 10mTalk | Test-based Patch Clustering for Automatically-Generated Patches Assessment Journal-First Matias Martinez Universitat Politècnica de Catalunya (UPC), Maria Kechagia National and Kapodistrian University of Athens, Anjana Perera Oracle Labs, Australia, Justyna Petke University College London, Federica Sarro University College London, Aldeida Aleti Monash University | ||
11:30 10mTalk | Hierarchical Knowledge Injection for Improving LLM-based Program Repair Research Papers Ramtin Ehsani Drexel University, Esteban Parra Rodriguez Belmont University, Sonia Haiduc Florida State University, Preetha Chatterjee Drexel University, USA | ||
11:40 10mTalk | Characterizing Multi-Hunk Patches: Divergence, Proximity, and LLM Repair Challenges Research Papers Noor Nashid University of British Columbia, Daniel Ding University of British Columbia, Keheliya Gallaba Centre for Software Excellence, Ahmed E. Hassan Queen’s University, Ali Mesbah University of British Columbia Pre-print | ||
11:50 10mTalk | Reinforcement Learning for Mutation Operator Selection in Automated Program Repair Journal-First Carol Hanna University College London, Aymeric Blot University of Rennes, IRISA / INRIA, Justyna Petke University College London | ||
12:00 10mTalk | Seeing is Fixing: Cross-Modal Reasoning with Multimodal LLMs for Visual Software Issue Repair Research Papers Kai Huang Technical University of Munich, Jian Zhang Nanyang Technological University, Xiaofei Xie Singapore Management University, Chunyang Chen TU Munich | ||