Automated Generation of Issue-Reproducing Tests by Combining LLMs and Search-Based Testing
This program is tentative and subject to change.
When a software issue is patched, it is important to have a test validating that the patch resolves the issue. Such a test fails in the unpatched code and passes in the patched code, hence called an issue-reproducing test. However, as with testing in general, the writing of issue-reproducing tests is frequently omitted by developers, making its automation an area of interest. We propose BLAST, a tool for automatically generating issue- reproducing tests from issue-patch pairs by combining LLMs and search-based software testing (SBST). For the LLM part, we complement the issue description and the patch by extracting relevant context through git history analysis, static analysis, and SBST-generated tests. For the SBST part, we adapt SBST for generating issue-reproducing tests; the issue description and the patch are fed into the SBST optimization through an interme- diate LLM-generated seed, which we deserialiaze into SBST- compatible form. BLAST successfully generates issue-reproducing tests for 151/426 (35.4%) of the issues from a curated Python benchmark, outperforming the state-of-the-art (23.5%). Additionally, to measure the real-world impact of BLAST, we built a GitHub bot that runs BLAST whenever a new pull request (PR) linked to an issue is opened, and if BLAST generates an issue- reproducing test, the bot proposes it as a comment in the PR. We deployed the bot in three open-source repositories for three months, gathering data for 32 PRs and proposing tests in 11 of them. By analyzing developers’ feedback, we discuss challenges and opportunities for researchers and tool builders.
This program is tentative and subject to change.
Wed 19 NovDisplayed time zone: Seoul change
14:00 - 15:30 | |||
14:00 10mTalk | LLMs for Automated Unit Test Generation and Assessment in Java: The AgoneTest Framework Research Papers Andrea Lops Polytechnic University of Bari, Italy, Fedelucio Narducci Polytechnic University of Bari, Azzurra Ragone University of Bari, Michelantonio Trizio Wideverse, Claudio Bartolini Wideverse s.r.l. | ||
14:10 10mTalk | µOpTime: Statically Reducing the Execution Time of Microbenchmark Suites Using Stability Metrics Journal-First Track Nils Japke TU Berlin & ECDF, Martin Grambow TU Berlin & ECDF, Christoph Laaber Simula Research Laboratory, David Bermbach TU Berlin | ||
14:20 10mTalk | Reference-Based Retrieval-Augmented Unit Test Generation Journal-First Track Zhe Zhang Beihang University, Liu Xingyu Beihang University, Yuanzhang Lin Beihang University, Xiang Gao Beihang University, Hailong Sun Beihang University, Yuan Yuan Beihang University | ||
14:30 10mTalk | Using Active Learning to Train Predictive Mutation Testing with Minimal Data Research Papers Miklos Borsi Karlsruhe Institute of Technology | ||
14:40 10mTalk | Clarifying Semantics of In-Context Examples for Unit Test Generation Research Papers Chen Yang Tianjin University, Lin Yang Tianjin University, Ziqi Wang Tianjin University, Dong Wang Tianjin University, Jianyi Zhou Huawei Cloud Computing Technologies Co., Ltd., Junjie Chen Tianjin University | ||
14:50 10mTalk | An empirical study of test case prioritization on the Linux Kernel Journal-First Track Haichi Wang College of Intelligence and Computing, Tianjin University, Ruiguo Yu College of Intelligence and Computing, Tianjin University, Dong Wang Tianjin University, Yiheng Du College of Intelligence and Computing, Tianjin University, Yingquan Zhao Tianjin University, Junjie Chen Tianjin University, Zan Wang Tianjin University | ||
15:00 10mTalk | Automated Generation of Issue-Reproducing Tests by Combining LLMs and Search-Based Testing Research Papers Konstantinos Kitsios University of Zurich, Marco Castelluccio Mozilla, Alberto Bacchelli University of Zurich Pre-print | ||
15:10 10mTalk | Using Fourier Analysis and Mutant Clustering to Accelerate DNN Mutation Testing Research Papers | ||
15:20 10mTalk | WEST: Specification-Based Test Generation for WebAssembly Research Papers | ||