PALM: Synergizing Program Analysis and LLMs to Enhance Rust Unit Test Coverage
This program is tentative and subject to change.
Unit testing is essential for ensuring software reliability and correctness. Classic Search-Based Software Testing (SBST) methods and concolic execution-based approaches for generating unit tests often fail to achieve high coverage due to difficulties in handling complex program units, such as branching conditions and external dependencies. Recent work has increasingly utilized large language models (LLMs) to generate test cases, improving the quality of test generation by providing better context and correcting errors in the model’s output. However, these methods rely on fixed prompts, resulting in relatively low compilation success rates and coverage.
This paper presents PALM, an approach that leverages large language models (LLMs) to enhance the generation of high-coverage unit tests. PALM performs program analysis to identify branching conditions within functions, which are then combined into path constraints. These constraints and relevant contextual information are used to construct prompts that guide the LLMs in generating unit tests. We implement the approach and evaluate it in 15 open-source Rust crates. Experimental results show that within just two or three hours, PALM can significantly improve test coverage compared to classic methods, with increases in overall project coverage exceeding 50% in some instances and its generated tests achieving an average coverage of 72.30%, comparable to human effort (70.94%), highlighting the potential of LLMs in automated test generation. We submitted 91 PALM-generated unit tests targeting new code. Of these submissions, 80 were accepted, 5 were rejected, and 6 remain pending review. The results demonstrate the effectiveness of integrating program analysis with AI and open new avenues for future research in automated software testing.
This program is tentative and subject to change.
Wed 19 NovDisplayed time zone: Seoul change
11:00 - 12:30 | |||
11:00 10mTalk | PALM: Synergizing Program Analysis and LLMs to Enhance Rust Unit Test Coverage Research Papers | ||
11:10 10mTalk | ROR-DSE: ROR adequate test case generation using dynamic symbolic execution Journal-First Track Sangharatna Godboley NIT Warangal | ||
11:20 10mTalk | Reflective Unit Test Generation for Precise Type Error Detection with Large Language Models Research Papers Chen Yang Tianjin University, Ziqi Wang Tianjin University, Yanjie Jiang Peking University, Lin Yang Tianjin University, Yuteng Zheng Tianjin University, Jianyi Zhou Huawei Cloud Computing Technologies Co., Ltd., Junjie Chen Tianjin University | ||
11:30 10mTalk | FailMapper: Automated Generation of Unit Tests Guided by Failure Scenarios Research Papers ruiqi dong Swinburne University of Technology, Zehang Deng Swinburne University of Technology, Xiaogang Zhu The University of Adelaide, Xiaoning Du Monash University, Huai Liu Swinburne University of Technology, Shaohua Wang Central University of Finance and Economics, Sheng Wen Swinburne University of Technology, Yang Xiang Digital Research & Innovation Capability Platform, Swinburne University of Technology | ||
11:40 10mTalk | Advancing Code Coverage: Incorporating Program Analysis with Large Language Models Journal-First Track Chen Yang Tianjin University, Junjie Chen Tianjin University, Bin Lin Hangzhou Dianzi University, Ziqi Wang Tianjin University, Jianyi Zhou Huawei Cloud Computing Technologies Co., Ltd. | ||
11:50 10mTalk | Navigating the Labyrinth: Path-Sensitive Unit Test Generation with Large Language Models Research Papers Dianshu Liao the Australian National University, Xin Yin Zhejiang University, Shidong Pan Columbia University & New York University, Chao Ni Zhejiang University, Zhenchang Xing CSIRO's Data61, Xiaoyu Sun Australian National University, Australia Pre-print | ||
12:00 10mTalk | Enhancing LLM’s Ability to Generate More Repository-Aware Unit Tests Through Precise Context Injection Research Papers Xin Yin Zhejiang University, Chao Ni Zhejiang University, Xinrui Li School of Software Technology, Zhejiang University, Liushan Chen Douyin Co., Ltd., Guojun Ma Douyin Co., Ltd., Xiaohu Yang Zhejiang University Pre-print | ||
12:10 10mTalk | Toward Cost-Effective Adaptive Random Testing: An Approximate Nearest Neighbor Approach Journal-First Track Rubing Huang Macau University of Science and Technology (M.U.S.T.), Chenhui Cui Macau University of Science and Technology, Junlong Lian Jiangsu University, Haibo Chen Jiangsu University, Dave Towey University of Nottingham Ningbo China, Weifeng Sun | ||
12:20 10mTalk | Automated Combinatorial Test Generation for Alloy Research Papers Agustín Borda Dept. of Computer Science FCEFQyN, University of Rio Cuarto, Germán Regis University of Rio Cuarto and CONICET, Nazareno Aguirre University of Rio Cuarto/CONICET, Argentina, and Guangdong Technion-Israel Institute of Technology, China, Marcelo F. Frias Dept. of Software Engineering Instituto Tecnológico de Buenos Aires, Pablo Ponzio Dept. of Computer Science FCEFQyN, University of Rio Cuarto | ||