FSE 2025
Mon 23 - Fri 27 June 2025 Trondheim, Norway
co-located with ISSTA 2025
Tue 24 Jun 2025 14:00 - 14:20 at Cosmos 3C - Testing 3 Chair(s): Dan Hao

Modern web services rely heavily on REST APIs, typically documented using the OpenAPI specification. The widespread adoption of this standard has resulted in the development of many black-box testing tools that generate tests based on these specifications. Recent advancements in Natural Language Processing (NLP), particularly with Large Language Models (LLMs), have enhanced REST API testing by extracting actionable rules and generating input values from the human-readable portions of the specification. However, these advancements overlook the potential of continuously refining the identified rules and test inputs based on server responses. To address this limitation, we present LlamaRestTest, a novel approach that employs two custom LLMs to generate realistic test inputs and uncover parameter dependencies during the testing process by incorporating server responses. These LLMs are created by fine-tuning the Llama3-8b model, using mined datasets of REST API example values and inter-parameter dependencies. We evaluated LlamaRestTest on 12 real-world services (including popular services such as Spotify), comparing it against RESTGPT, a GPT-powered specification-enhancement tool, as well as several state-of-the-art REST API testing tools, including RESTler, MoRest, EvoMaster, and ARAT-RL. Our results demonstrate that fine-tuning enables smaller LLMs to outperform much larger models in detecting actionable rules and generating inputs for REST API testing. We also evaluated different tool configurations, ranging from the base Llama3-8B model to fine-tuned versions tailored for REST API testing, and explored multiple quantization techniques, including 2-bit, 4-bit, and 8-bit integer formats, to improve efficiency and performance. Our study shows that small language models can perform similar to, or better than, large language models in REST API testing, striking a balance between effectiveness and efficiency. Furthermore, LlamaRestTest outperforms state-of-the-art REST API testing tools in code coverage achieved and internal server errors identified, even when those tools utilize enhanced specifications generated by RESTGPT. Finally, through an ablation study, we show that each of the novel components of LlamaRestTest contributes to its overall performance.

Tue 24 Jun

Displayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change

14:00 - 15:30
14:00
20m
Talk
LlamaRestTest: Effective REST API Testing with Small Language Models
Research Papers
Myeongsoo Kim Georgia Institute of Technology, Saurabh Sinha IBM Research, Alessandro Orso Georgia Institute of Technology
DOI
14:20
20m
Talk
Testing Updated Apps by Adapting Learned Models
Journal First
Chanh Duc Ngo University of Luxembourg, Fabrizio Pastore University of Luxembourg, Lionel Briand University of Ottawa, Canada; Lero centre, University of Limerick, Ireland
Link to publication
14:40
20m
Talk
Automated Testing of COBOL to Java Transformation
Industry Papers
Sandeep Hans IBM India Research Lab, Atul Kumar IBM Research India, Toshiaki Yasue IBM Research - Tokyo, Kohichi Ono IBM Research - Tokyo, Saravanan Krishnan IBM India Research Lab, Devika Sondhi IBM Research, Fumiko Satoh IBM Research - Tokyo, Gerald Mitchell IBM Software, Sachin Kumar IBM Software, Diptikalyan Saha IBM Research India
15:00
20m
Talk
TerzoN: Human-in-the-Loop Software Testing with a Composite Oracle
Research Papers
Matthew C. Davis Carnegie Mellon University, Amy Wei University of Michigan, Brad A. Myers Carnegie Mellon University, Joshua Sunshine Carnegie Mellon University
Link to publication DOI
15:20
10m
Talk
Efficient Test Generation for Dynamic Behaviors Leveraging Token-Level Input Commonalities
Ideas, Visions and Reflections
Yuxin Qiu University of California at Riverside, Qian Zhang University of California at Riverside

Information for Participants
Tue 24 Jun 2025 14:00 - 15:30 at Cosmos 3C - Testing 3 Chair(s): Dan Hao
Info for room Cosmos 3C:

Cosmos 3C is the third room in the Cosmos 3 wing.

When facing the main Cosmos Hall, access to the Cosmos 3 wing is on the left, close to the stairs. The area is accessed through a large door with the number “3”, which will stay open during the event.

:
:
:
: