LlamaRestTest: Effective REST API Testing with Small Language Models
Modern web services rely heavily on REST APIs, typically documented using the OpenAPI specification. The widespread adoption of this standard has resulted in the development of many black-box testing tools that generate tests based on these specifications. Recent advancements in Natural Language Processing (NLP), particularly with Large Language Models (LLMs), have enhanced REST API testing by extracting actionable rules and generating input values from the human-readable portions of the specification. However, these advancements overlook the potential of continuously refining the identified rules and test inputs based on server responses. To address this limitation, we present LlamaRestTest, a novel approach that employs two custom LLMs to generate realistic test inputs and uncover parameter dependencies during the testing process by incorporating server responses. These LLMs are created by fine-tuning the Llama3-8b model, using mined datasets of REST API example values and inter-parameter dependencies. We evaluated LlamaRestTest on 12 real-world services (including popular services such as Spotify), comparing it against RESTGPT, a GPT-powered specification-enhancement tool, as well as several state-of-the-art REST API testing tools, including RESTler, MoRest, EvoMaster, and ARAT-RL. Our results demonstrate that fine-tuning enables smaller LLMs to outperform much larger models in detecting actionable rules and generating inputs for REST API testing. We also evaluated different tool configurations, ranging from the base Llama3-8B model to fine-tuned versions tailored for REST API testing, and explored multiple quantization techniques, including 2-bit, 4-bit, and 8-bit integer formats, to improve efficiency and performance. Our study shows that small language models can perform similar to, or better than, large language models in REST API testing, striking a balance between effectiveness and efficiency. Furthermore, LlamaRestTest outperforms state-of-the-art REST API testing tools in code coverage achieved and internal server errors identified, even when those tools utilize enhanced specifications generated by RESTGPT. Finally, through an ablation study, we show that each of the novel components of LlamaRestTest contributes to its overall performance.
Tue 24 JunDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
14:00 - 15:30 | Testing 3Research Papers / Industry Papers / Ideas, Visions and Reflections / Journal First at Cosmos 3C Chair(s): Dan Hao Peking University | ||
14:00 20mTalk | LlamaRestTest: Effective REST API Testing with Small Language Models Research Papers Myeongsoo Kim Georgia Institute of Technology, Saurabh Sinha IBM Research, Alessandro Orso Georgia Institute of Technology DOI | ||
14:20 20mTalk | Testing Updated Apps by Adapting Learned Models Journal First Chanh Duc Ngo University of Luxembourg, Fabrizio Pastore University of Luxembourg, Lionel Briand University of Ottawa, Canada; Lero centre, University of Limerick, Ireland Link to publication | ||
14:40 20mTalk | Automated Testing of COBOL to Java Transformation Industry Papers Sandeep Hans IBM India Research Lab, Atul Kumar IBM Research India, Toshiaki Yasue IBM Research - Tokyo, Kohichi Ono IBM Research - Tokyo, Saravanan Krishnan IBM India Research Lab, Devika Sondhi IBM Research, Fumiko Satoh IBM Research - Tokyo, Gerald Mitchell IBM Software, Sachin Kumar IBM Software, Diptikalyan Saha IBM Research India | ||
15:00 20mTalk | TerzoN: Human-in-the-Loop Software Testing with a Composite Oracle Research Papers Matthew C. Davis Carnegie Mellon University, Amy Wei University of Michigan, Brad A. Myers Carnegie Mellon University, Joshua Sunshine Carnegie Mellon University Link to publication DOI | ||
15:20 10mTalk | Efficient Test Generation for Dynamic Behaviors Leveraging Token-Level Input Commonalities Ideas, Visions and Reflections |
Cosmos 3C is the third room in the Cosmos 3 wing.
When facing the main Cosmos Hall, access to the Cosmos 3 wing is on the left, close to the stairs. The area is accessed through a large door with the number “3”, which will stay open during the event.