Evaluating the Impact of Experimental Assumptions in Automated Fault Localization
Much research on automated program debugging often assumes that bug fix location(s) indicate the faults’ root causes and that root causes of faults lie within single code elements (statements). It is also often assumed that the number of statements a developer would need to inspect before finding the first faulty statement reflects debugging effort. Although intuitive, these three assumptions are typically used (55% of experiments in surveyed publications make at least one of these three assumptions) without any consideration of their effects on the debugger’s effectiveness and potential impact on developers in practice. To deal with this issue, we perform controlled experimentation, split testing in particular, using 352 bugs from 46 open-source C programs, 19 Automated Fault Localization (AFL) techniques (18 statistical debugging formulas and dynamic slicing), two (2) state-of-the-art automated program repair (APR) techniques (GenProg and Angelix) and 76 professional developers. Our results show that these assumptions conceal the difficulty of debugging. They make AFL techniques appear to be (up to 38%) more effective, and make APR tools appear to be (2X) less effective. We also find that most developers (83%) consider these assumptions to be unsuitable for debuggers and, perhaps worse, that they may inhibit development productivity. The majority (66%) of developers prefer debugging diagnoses without these assumptions twice as much as with the assumptions. Our findings motivate the need to assess debuggers conservatively, i.e., without these assumptions.
Wed 17 MayDisplayed time zone: Hobart change
11:00 - 12:30 | Fault localizationJournal-First Papers / Technical Track / Showcase at Meeting Room 103 Chair(s): Rui Abreu University of Porto | ||
11:00 15mTalk | Evaluating the Impact of Experimental Assumptions in Automated Fault Localization Technical Track Ezekiel Soremekun Royal Holloway, University of London, Lukas Kirschner Saarland University, Marcel Böhme MPI-SP, Germany and Monash University, Australia, Mike Papadakis University of Luxembourg, Luxembourg Pre-print Media Attached | ||
11:15 15mTalk | Locating Framework-specific Crashing Faults with Compact and Explainable Candidate Set Technical Track Jiwei Yan Institute of Software at Chinese Academy of Sciences, China, MiaoMiao Wang Technology Center of Software Engineering, ISCAS, China. University of Chinese Academy of Sciences, China., Yepang Liu Southern University of Science and Technology, Jun Yan Institute of Software at Chinese Academy of Sciences; University of Chinese Academy of Sciences, Long Zhang Institute of Software, Chinese Academy of Sciences Pre-print | ||
11:30 15mTalk | PExReport: Automatic Creation of Pruned Executable Cross-Project Failure Reports Technical Track Pre-print Media Attached | ||
11:45 15mTalk | Bug localization in game software engineering: evolving simulations to locate bugs in software models of video games Showcase Rodrigo Casamayor SVIT Research Group. Universidad San Jorge, Lorena Arcega San Jorge University, Francisca Pérez SVIT Research Group, Universidad San Jorge, Carlos Cetina San Jorge University, Spain DOI | ||
12:00 7mTalk | Real World Projects, Real Faults: Evaluating Spectrum Based Fault Localization Techniques on Python Projects Journal-First Papers Ratnadira Widyasari Singapore Management University, Singapore, Gede Artha Azriadi Prana Singapore Management University, Stefanus Agus Haryono Singapore Management University, Shaowei Wang University of Manitoba, David Lo Singapore Management University | ||
12:07 7mTalk | Effective Isolation of Fault-Correlated Variables via Statistical and Mutation Analysis Journal-First Papers Ming Wen Huazhong University of Science and Technology, Zifan Xie Huazhong University of Science and Technology, Kaixuan Luo Huazhong University of Science and Technology, Xiao Chen Huazhong University of Science and Technology, Yibiao Yang Nanjing University, Hai Jin Huazhong University of Science and Technology | ||
12:15 15mTalk | RAT: A Refactoring-Aware Traceability Model for Bug Localization Technical Track Feifei Niu University of Ottawa, Wesley Assunção Johannes Kepler University Linz, Austria & Pontifical Catholic University of Rio de Janeiro, Brazil, Liguo Huang Southern Methodist University, Christoph Mayr-Dorn JOHANNES KEPLER UNIVERSITY LINZ, Jidong Ge Nanjing University, Bin Luo Nanjing University, Alexander Egyed Johannes Kepler University Linz File Attached |