On the Efficiency of Test Suite based Program Repair: A Systematic Assessment of 16 Automated Repair Systems for Java ProgramsTechnical
Test-based automated program repair has been a prolific field of research in software engineering in the last decade. Many approaches have indeed been proposed, which leverage test suites as a weak, but affordable, approximation to program specifications. While the literature regularly sets new records on the number of benchmark bugs that can be fixed, several studies increasingly raise concerns about the limitations and biases of state-of-the-art approaches. For example, the correctness of generated patches has been questioned in a number of studies, while other researchers pointed out that evaluation schemes may be misleading with respect to the processing of fault localization results. Nevertheless, there is little work addressing the efficiency of patch generation, with regard to the practicality of program repair. In this paper, we fill this gap in the literature, by providing an extensive review on the efficiency of test suite based program repair. We focus on assessing the number of generated patch candidates, since this information is correlated to (1) the strategy to traverse the search space efficiently in order to select sensical repair attempts, (2) the strategy to minimize the test effort for identifying a plausible patch, (3) as well as the strategy to prioritize the generation of a correct patch. To that end, we perform a large-scale empirical study on the efficiency, in terms of quantity of generated patch candidates of the 16 open-source repair tools for Java programs. The experiments are carefully run under the same fault localization configurations to limit biases. Eventually, among other findings, we note that: (1) many irrelevant patch candidates are generated by APR tools by changing wrong code locations; (2) however, if the search space is carefully triaged, fault localization noise has little impact on patch generation efficiency; (3) yet, current template-based repair systems, which are known to be most effective in fixing a large number of bugs, are actually least efficient as they tend to generate majoritarily irrelevant patch candidates.
Wed 8 JulDisplayed time zone: (UTC) Coordinated Universal Time change
00:00 - 01:00 | P9-Bugs and RepairJournal First / Technical Papers / Software Engineering in Practice / New Ideas and Emerging Results at Silla Chair(s): Yingfei Xiong Peking University, China | ||
00:00 12mTalk | PRECFIX: Large-Scale Patch Recommendation by Mining Defect-Patch PairsSEIP Software Engineering in Practice Xindong Zhang Alibaba Group, Chenguang Zhu University of Texas, Austin, Yi Li Nanyang Technological University, Jianmei Guo Alibaba Group, Lihua Liu Alibaba Group, Haobo Gu Alibaba Group Pre-print | ||
00:12 12mTalk | On the Efficiency of Test Suite based Program Repair: A Systematic Assessment of 16 Automated Repair Systems for Java ProgramsTechnical Technical Papers Kui Liu Huawei Software Engineering Application Technology Lab, Shangwen Wang National University of Defense Technology, Anil Koyuncu University of Luxembourg, Luxembourg, Kisub Kim University of Luxembourg, SnT, Tegawendé F. Bissyandé SnT, University of Luxembourg, Dongsun Kim Furiosa.ai, Peng Wu National University of Defense Technology, Jacques Klein University of Luxembourg, SnT, Xiaoguang Mao National University of Defense Technology, Yves Le Traon University of Luxembourg Pre-print | ||
00:24 8mTalk | SEQUENCER: Sequence-to-Sequence Learning for End-to-End Program RepairJ1 Journal First Zimin Chen KTH Royal Institute of Technology, Steve Kommrusch Colorado State University, Michele Tufano College of William and Mary, Louis-Noël Pouchet Colorado State University, USA, Denys Poshyvanyk William and Mary, Martin Monperrus KTH Royal Institute of Technology | ||
00:32 8mTalk | A Study of Bug Resolution Characteristics in Popular Programming LanguagesJ1 Journal First Jie M. Zhang University College London, UK, Feng Li , Dan Hao Peking University, Meng Wang University of Bristol, UK, Hao Tang Peking University, Lu Zhang Peking University, China, Mark Harman Facebook and University College London | ||
00:40 12mTalk | Automated Bug Reproduction from User Reviews for Android ApplicationsSEIP Software Engineering in Practice Shuyue Li Xi'an Jiaotong University, Jiaqi Guo Xi'an Jiaotong University, Ming Fan Xi'an Jiaotong University, Jian-Guang Lou Microsoft Research, Qinghua Zheng Xi'an Jiaotong University, Ting Liu Xi'an Jiaotong University | ||
00:52 6mTalk | CHASE: Checklist to Assess User Experience in Internet of Things EnvironmentsNIER New Ideas and Emerging Results Rodrigo Almeida Federal University of Ceará, Joseane Paiva Federal University of Ceará, Rossana Andrade Federal University of Ceará, Ticianne Darin Federal University of Ceará |