Shibboleth: Hybrid Patch Correctness Assessment in Automated Program Repair
Wed 12 Oct 2022 16:00 - 16:10 at Banquet A - Technical Session 18 - Testing II Chair(s): Darko Marinov
Test-based generate-and-validate automated program repair (APR) systems generate many patches that pass the test suite without fixing the bug. The generated patches must be manually inspected by the developers, a task that tends to be time-consuming, thereby diminishing the role of APR in reducing debugging costs. We present the design and implementation of a novel tool, named Shibboleth, for automatic assessment of the patches generated by test-based generate-and-validate APR systems. Shibboleth leverages lightweight static and dynamic heuristics from both test and production code to rank and classify the patches. Shibboleth is based on the idea that the buggy program is almost correct and the bugs are small mistakes that require small changes to fix and specifically the fix does not remove the code implementing correct functionality of the program. Thus, the tool measures the impact of patches on both production code (via syntactic and semantic similarity) and test code (via code coverage) to separate the patches that result in similar programs and that do not remove desired program elements. We have evaluated Shibboleth on 1,871 patches, generated by 29 Java-based APR systems for Defects4J programs. The technique outperforms state-of-the-art raking and classification techniques. Specifically, in our ranking data set, in 66% of the cases, Shibboleth ranks the correct patch in top-1 or top-2 positions and, in our classification data set, it achieves an accuracy and F1-score of 0.887 and 0.852, respectively, in classification mode. A demo video of the tool is available at https://bit.ly/3NvYJN8.
Tue 11 OctDisplayed time zone: Eastern Time (US & Canada) change
10:00 - 10:30 | |||
10:00 30mDemonstration | WebMonitor: https://youtu.be/hqVw0JU3k9c Tool Demonstrations Ennio Visconti TU Wien, Christos Tsigkanos University of Bern, Switzerland, Laura Nenzi University of Trieste | ||
10:00 30mDemonstration | Quacky: Quantitative Access Control Permissiveness Analyzer Tool Demonstrations William Eiers University of California at Santa Barbara, USA, Ganesh Sankaran University of California Santa Barbara, Albert Li University of California Santa Barbara, Emily O'Mahony University of California Santa Barbara, Benjamin Prince University of California Santa Barbara, Tevfik Bultan University of California, Santa Barbara | ||
10:00 30mDemonstration | Snapshot Metrics Are Not Enough: Analyzing Software Repositories with Longitudinal Metrics Tool Demonstrations Nicholas Synovic Loyola University Chicago, Matt Hyatt Loyola University Chicago, Rohan Sethi Loyola University Chicago, Sohini Thota Loyola University Chicago, Shilpika University of California at Davis, Allan J. Miller Loyola University Chicago, Wenxin Jiang Purdue University, Emmanuel S. Amobi Loyola University Chicago, Austin Pinderski Duke University, Loyola University Chicago, Konstantin Läufer Loyola University Chicago, Nicholas J. Hayward Loyola University Chicago, Neil Klingensmith Loyola University Chicago, James C. Davis Purdue University, USA, George K. Thiruvathukal Loyola University Chicago and Argonne National Laboratory | ||
10:00 30mDemonstration | AUSERA: Automated Security Vulnerability Detection for Android AppsVirtual Tool Demonstrations Sen Chen Tianjin University, Yuxin Zhang Tianjin University, Lingling Fan Nankai University, Jiaming Li Tianjin University, Yang Liu Nanyang Technological University | ||
10:00 30mDemonstration | Trimmer: Context-Specific Code ReductionVirtual Tool Demonstrations Aatira Anum Ahmad Lahore University of Management Sciences, Mubashir Anwar University of Illinois Urbana-Champaign, Hashim Sharif University of Illinois at Urbana-Champaign, Ashish Gehani SRI, Fareed Zaffar Lahore University of Management Sciences | ||
10:00 30mDemonstration | Maktub: Lightweight Robot System Test Creation and Automation Tool Demonstrations | ||
10:00 30mDemonstration | V-Achilles: An Interactive Visualization of Transitive Security Vulnerabilities Tool Demonstrations Vipawan Jarukitpipat Mahidol University, Xiao Peng China EverBright Bank, Xiao Peng China EverBright Bank, Chaiyong Ragkhitwetsagul Mahidol University, Thailand, Morakot Choetkiertikul Mahidol University, Thailand, Thanwadee Sunetnanta Mahidol University, Raula Gaikovina Kula Nara Institute of Science and Technology, Bodin Chinthanet Nara Institute of Science and Technology, Takashi Ishio Nara Institute of Science and Technology, Kenichi Matsumoto Nara Institute of Science and Technology | ||
10:00 30mDemonstration | RobSimVer: A Tool for RoboSim Modeling and AnalysisVirtual Tool Demonstrations Dehui Du East China Normal University, Ana Cavalcanti University of York, JihuiNie East China Normal University | ||
10:00 30mDemonstration | Xscope: Hunting for Cross-Chain Bridge AttacksVirtual Tool Demonstrations Jiashuo Zhang Peking University, China, Jianbo Gao Peking University, Yue Li Peking University, Ziming Chen Peking University, Zhi Guan Peking University, Zhong Chen | ||
10:00 30mDemonstration | SAFA: A Tool for Supporting Safety Analysis in Evolving Software Systems Tool Demonstrations Alberto D. Rodriguez University of Notre Dame, Timothy Newman University of Notre Dame, Katherine R. Dearstyne University of Notre Dame, Jane Cleland-Huang University of Notre Dame | ||
10:00 30mDemonstration | Building recommender systems for modelling languages with DroidVirtual Tool Demonstrations Lissette Almonte Universidad Autónoma de Madrid, Esther Guerra Universidad Autónoma de Madrid, Iván Cantador Universidad Autónoma de Madrid, Juan de Lara Autonomous University of Madrid Pre-print Media Attached | ||
10:00 30mDemonstration | Shibboleth: Hybrid Patch Correctness Assessment in Automated Program Repair Tool Demonstrations |
Wed 12 OctDisplayed time zone: Eastern Time (US & Canada) change
16:00 - 18:00 | Technical Session 18 - Testing IIResearch Papers / Tool Demonstrations / Journal-first Papers at Banquet A Chair(s): Darko Marinov University of Illinois at Urbana-Champaign | ||
16:00 10mDemonstration | Shibboleth: Hybrid Patch Correctness Assessment in Automated Program Repair Tool Demonstrations | ||
16:10 20mResearch paper | Auto Off-Target: Enabling Thorough and Scalable Testing for Complex Software Systems Research Papers DOI Pre-print | ||
16:30 10mDemonstration | Maktub: Lightweight Robot System Test Creation and Automation Tool Demonstrations | ||
16:40 20mPaper | Cerebro: Static Subsuming Mutant Selection Journal-first Papers Aayush Garg University of Luxembourg, Milos Ojdanic University of Luxembourg, Renzo Degiovanni SnT, University of Luxembourg, Thierry Titcheu Chekam SES S.A. & University of Luxembourg (SnT), Mike Papadakis University of Luxembourg, Luxembourg, Yves Le Traon University of Luxembourg, Luxembourg Link to publication DOI | ||
17:00 20mResearch paper | Natural Test Generation for Precise Testing of Question Answering SoftwareVirtual Research Papers Qingchao Shen Tianjin University, Junjie Chen Tianjin University, Jie M. Zhang King's College London, Haoyu Wang College of Intelligence and Computing, Tianjin University, Shuang Liu Tianjin University, Menghan Tian College of Intelligence and Computing, Tianjin University Pre-print | ||
17:20 20mPaper | GloBug: Using global data in Fault LocalizationVirtual Journal-first Papers Nima Miryeganeh University of Calgary, Sepehr Hashtroudi University of Calgary, Hadi Hemmati University of Calgary Link to publication DOI | ||
17:40 20mResearch paper | Selectively Combining Multiple Coverage Goals in Search-Based Unit Test GenerationVirtual Research Papers Zhichao Zhou School of Information Science and Technology, ShanghaiTech University, Yuming Zhou Nanjing University, Chunrong Fang Nanjing University, Zhenyu Chen Nanjing University, Yutian Tang ShanghaiTech University DOI Pre-print |