FailMapper: Automated Generation of Unit Tests Guided by Failure Scenarios
The automation of unit test generation has become a critical task for improving the overall efficiency of software development testing. Many existing techniques attempt to generate a sufficient number of test cases to achieve high code coverage. However, it has been shown that a high coverage does not necessarily guarantee effective bug discovery. A potential enhancement is to guide the unit test generation based on bug properties. However, this solution is challenged by the large number and diversity of bug types, making it difficult to comprehensively summarize bug properties.
We observe that, failures, presented as the results of bugs, manifest in a limited number of scenarios. Therefore, instead of bug properties, in this paper, we propose an innovative framework, named FailMapper, which uses failure scenarios to guide the generation of unit tests. We summarize nine failure scenarios, and design the corresponding failure-triggering test strategies. This significantly improves the efficacy of generating test cases towards triggering bugs. To systematically explore possible failure scenarios, FailMapper employs the Monte Carlo Tree Search (MCTS) algorithm to search the faults that may lead to a failure. Experiments demonstrate that, on 50 known bugs in the Defects4J benchmark, FailMapper can detect much more bugs than five typical unit testing approaches, including EvoSuite, Randoop, CoverUp, HITS, and SymPrompt (40 versus at most 12, out of all 50 bugs). FailMapper can also reveal 36 previously undiscovered bugs, further demonstrating its effectiveness. The experimental results clearly show that our new framework can significantly enhance the overall efficacy of unit testing.
Wed 19 NovDisplayed time zone: Seoul change
11:00 - 12:30 | Test Generation, Selection & Prioritization 1Research Papers / Journal-First at Vista Chair(s): Owolabi Legunsen Cornell University | ||
11:00 10mTalk | PALM: Synergizing Program Analysis and LLMs to Enhance Rust Unit Test Coverage Research Papers | ||
11:10 10mTalk | ROR-DSE: ROR adequate test case generation using dynamic symbolic execution Journal-First Sangharatna Godboley NIT Warangal | ||
11:20 10mTalk | Reflective Unit Test Generation for Precise Type Error Detection with Large Language Models Research Papers Chen Yang Tianjin University, Ziqi Wang Tianjin University, Yanjie Jiang Peking University, Lin Yang Tianjin University, Yuteng Zheng Tianjin University, Jianyi Zhou Huawei Cloud Computing Technologies Co., Ltd., Junjie Chen Tianjin University | ||
11:30 10mTalk | FailMapper: Automated Generation of Unit Tests Guided by Failure Scenarios Research Papers ruiqi dong Swinburne University of Technology, Zehang Deng Swinburne University of Technology, Xiaogang Zhu Adelaide University, Xiaoning Du Monash University, Huai Liu Swinburne University of Technology, Shaohua Wang Central University of Finance and Economics, Sheng Wen Swinburne University of Technology, Yang Xiang Digital Research & Innovation Capability Platform, Swinburne University of Technology | ||
11:40 10mTalk | Advancing Code Coverage: Incorporating Program Analysis with Large Language Models Journal-First Chen Yang Tianjin University, Junjie Chen Tianjin University, Bin Lin Hangzhou Dianzi University, Ziqi Wang Tianjin University, Jianyi Zhou Huawei Cloud Computing Technologies Co., Ltd. | ||
11:50 10mTalk | Navigating the Labyrinth: Path-Sensitive Unit Test Generation with Large Language Models Research Papers Dianshu Liao the Australian National University, Xin Yin Zhejiang University, Shidong Pan Columbia University & New York University, Chao Ni Zhejiang University, Zhenchang Xing CSIRO's Data61, Xiaoyu Sun Australian National University, Australia Pre-print | ||
12:00 10mTalk | Enhancing LLM’s Ability to Generate More Repository-Aware Unit Tests Through Precise Context Injection Research Papers Xin Yin Zhejiang University, Chao Ni Zhejiang University, Xinrui Li Zhejiang University, Liushan Chen Douyin Co., Ltd., Guojun Ma Douyin Co., Ltd., Xiaohu Yang Zhejiang University Pre-print | ||
12:10 10mTalk | Toward Cost-Effective Adaptive Random Testing: An Approximate Nearest Neighbor Approach Journal-First Rubing Huang Macau University of Science and Technology (MUST), Chenhui Cui Macau University of Science and Technology, Junlong Lian Jiangsu University, Haibo Chen Jiangsu University, Dave Towey University of Nottingham Ningbo China, Weifeng Sun | ||
12:20 10mTalk | Automated Combinatorial Test Generation for Alloy Research Papers Agustín Borda University of Rio Cuarto, CONICET and Guangdong Technion-Israel Institute of Technology, Germán Regis University of Rio Cuarto and CONICET, Nazareno Aguirre University of Rio Cuarto/CONICET, Argentina, and Guangdong Technion-Israel Institute of Technology, China, Marcelo F. Frias Dept. of Software Engineering Instituto Tecnológico de Buenos Aires, Pablo Ponzio Dept. of Computer Science FCEFQyN, University of Rio Cuarto | ||