ISSTA 2025
Wed 25 - Sat 28 June 2025 Trondheim, Norway
Wed 25 Jun 2025 14:50 - 15:15 at Cosmos 3A - LLM-based Testing 1 Chair(s): Qingkai Shi

This paper presents Tratto, a neuro-symbolic approach that generates assertions (boolean expressions) that can serve as axiomatic oracles, from source code and documentation. The symbolic module of Tratto takes advantage of the grammar of the programming language, the unit under test, and the context of the unit (its class and available APIs) to restrict the search space of the tokens that can be successfully used to generate valid oracles. The neural module of Tratto uses transformers fine-tuned for both deciding whether to output an oracle or not and selecting the next lexical token to incrementally build the oracle from the set of tokens returned by the symbolic module. Our experiments show that Tratto outperforms the state-of-the-art axiomatic oracle generation approaches, with 73% accuracy, 72% precision, and 61% F1-score, largely higher than the best results of the symbolic and neural approaches considered in our study (61%, 62%, and 37%, respectively). Tratto can generate three times more axiomatic oracles than current symbolic approaches, while generating 10 times less false positives than GPT4 complemented with few-shot learning and Chain-of-Thought prompting.

Wed 25 Jun

Displayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change

14:00 - 15:30
LLM-based Testing 1Research Papers / Tool Demonstrations at Cosmos 3A
Chair(s): Qingkai Shi Nanjing University
14:00
25m
Talk
A Large-scale Empirical Study on Fine-tuning Large Language Models for Unit Testing
Research Papers
ye shang Nanjing University, Quanjun Zhang School of Computer Science and Engineering, Nanjing University of Science and Technology, Chunrong Fang Nanjing University, Siqi Gu Nanjing University, Jianyi Zhou Huawei Cloud Computing Technologies Co., Ltd., Zhenyu Chen Nanjing University
DOI
14:25
25m
Talk
Validating Network Protocol Parsers with Traceable RFC Document Interpretation
Research Papers
Mingwei Zheng Purdue University, Danning Xie Purdue University, Qingkai Shi Nanjing University, Chengpeng Wang Purdue University, Xiangyu Zhang Purdue University
DOI
14:50
25m
Talk
Tratto: A Neuro-Symbolic Approach to Deriving Axiomatic Test Oracles
Research Papers
Davide Molinelli USI Lugano; Schaffhausen Institute of Technology, Alberto Martin-Lopez Software Institute - USI, Lugano, Elliott Zackrone University of Washington, Beyza Eken Sakarya University, Michael D. Ernst University of Washington, Mauro Pezze Università della Svizzera italiana (USI) and Università degli Studi di Milano Bicocca and CIT Constructor Institute of Technology
DOI Pre-print
15:15
15m
Demonstration
Kitten: A Simple Yet Effective Baseline for Evaluating LLM-Based Compiler Testing Techniques
Tool Demonstrations
Yuanmin Xie Tsinghua University, Zhenyang Xu University of Waterloo, Yongqiang Tian , Min Zhou , Xintong Zhou University of Waterloo, Chengnian Sun University of Waterloo

Information for Participants
Wed 25 Jun 2025 14:00 - 15:30 at Cosmos 3A - LLM-based Testing 1 Chair(s): Qingkai Shi
Info for room Cosmos 3A:

Cosmos 3A is the first room in the Cosmos 3 wing.

When facing the main Cosmos Hall, access to the Cosmos 3 wing is on the left, close to the stairs. The area is accessed through a large door with the number “3”, which will stay open during the event.