SAINT: Service-level Integration Test Generation with Program Analysis and LLM-based Agents
Enterprise applications are typically tested at multiple levels, with service-level testing playing an important role in validating application functionality. Existing service-level testing tools, especially for RESTful APIs, often employ fuzzing and/or depend on OpenAPI specifications which are not readily available in real-world enterprise codebases. Moreover, they fail to generate functional tests that exercise meaningful scenarios. In this work, we present SAINT, a novel white-box testing approach for service-level testing of enterprise Java applications. SAINT combines static analysis, large language models (LLMs), and LLM-based agents to automatically generate endpoint and scenario-based tests. The approach builds two key models: an endpoint model, capturing syntactic and semantic information about service endpoints, and an operation dependency graph, capturing inter-endpoint ordering constraints. SAINT then employs LLM-based agents to generate tests. Endpoint-focused tests aim to maximize code and database interaction coverage. Scenario-based tests are synthesized by extracting application use cases from code and refining them into executable tests via planning, action, and reflection phases of the agentic loop. We evaluated SAINT on eight Java applications, including a proprietary enterprise application. Our results illustrate the effectiveness of SAINT in coverage, fault detection, and scenario generation. Moreover, a developer survey provides strong endorsement of the scenario-based tests generated by SAINT. Overall, our work shows that combining static analysis with agentic LLM workflows enables more effective, functional, and developer-aligned service-level test generation.
Thu 16 AprDisplayed time zone: Brasilia, Distrito Federal, Brazil change
14:00 - 15:30 | AI for Software Engineering 14Research Track at Asia IV Chair(s): Reyhaneh Jabbarvand University of Illinois at Urbana-Champaign | ||
14:00 15mTalk | Knowledge Matters: Injecting Project and Testing Knowledge into LLM-based Unit Test Generation Research Track Anji Li School of Software Engineering, Sun Yat-sen University, Mingwei Liu Sun Yat-Sen University, Zhenxi Chen Sun Yat-Sen University, Zheng Pei Sun Yat-Sen University, Zike Li Sun Yat-Sen University, Dekun Dai Sun Yat-Sen University, Yanlin Wang Sun Yat-sen University, Zibin Zheng Sun Yat-sen University DOI Pre-print Media Attached | ||
14:15 15mTalk | Issue2Test: Generating Reproducing Test Cases from Issue Reports Research Track Noor Nashid University of British Columbia, Islem BOUZENIA CISPA Helmholtz Center for Information Security, Michael Pradel CISPA Helmholtz Center for Information Security, Ali Mesbah University of British Columbia | ||
14:30 15mTalk | RBCTest: Leveraging LLMs to Mine and Verify Oracles of API Response Bodies for RESTful API Testing Research Track Hieu Huynh University of Melbourne, Quoc-Tri Le Katalon LLC, Tu Nguyen University of Science, VNU-HCM, Viet Nguyen University of Science, VNU-HCM, Vu Nguyen University of Science, VNU-HCM; Katalon LLC., Tien N. Nguyen University of Texas at Dallas | ||
14:45 15mTalk | Measuring the Influence of Incorrect Code on Test Generation Research Track Dong Huang The University of Hong Kong, Jie M. Zhang King's College London, Mark Harman Meta Platforms, Inc. and UCL, Mingzhe Du National University of Singapore, Heming Cui University of Hong Kong | ||
15:00 15mTalk | Retrieval-Augmented Test Generation: How Far Are We? Research Track Jiho Shin Queen's University, Nima Shiri Harzevili York University, Reem Aleithan York University, Canada, Hadi Hemmati York University, Song Wang York University Pre-print | ||
15:15 15mTalk | SAINT: Service-level Integration Test Generation with Program Analysis and LLM-based Agents Research Track Rangeet Pan IBM Research, Raju Pavuluri IBM T.J. Watson Research Center, Ruikai Huang Georgia Institute of Technology, Tyler Stennett Georgia Institute of Technology, Rahul Krishna IBM Research, Alessandro Orso University of Georgia, USA, Saurabh Sinha IBM Research | ||