Efficient Test Generation for Dynamic Behaviors Leveraging Token-Level Input Commonalities
Software in the wild exhibits various dynamic and nondeterministic behaviors, such as runtime invocation from dynamic language features and randomness from probabilistic algorithms. These behaviors complicate software testing because they require meaningful inputs for invocation, introduce additional branching decisions at runtime, and produce divergent outcomes for identical inputs.
To address these challenges, we propose an effective and efficient testing framework that leverages commonality analysis. This paper presents our preliminary research, which leverages input commonalities to explore dynamic behaviors in modern Java software, to demonstrate the utility of our framework. Specifically, we introduce SynToken, a two-fold testing tool that aims to improve (1) the reachability of runtime invocation sites and (2) testing efficiency. First, to address the challenge that existing string mutation operators cannot easily synthesize meaningful inputs for runtime invocation, SynToken constructs a token tree to prioritize tokens that are likely to be used for invocation based on common vocabularies such as prefixes or suffixes. Second, to address the challenge that branching decisions and execution paths are not deterministic, SynToken enhances traditional branch coverage guided fuzzing by monitoring invocation behaviors based on class hierarchies and call graphs.
We preliminarily evaluated SynToken on 71 open-source subjects. SynToken accelerated test generation by up to 10 × with tree-based input prioritization. The string substitution based input mutation enabled SynToken to generate all loadable targets within two minutes for each subject and replicated 280 injected bugs.
In the future, we plan to extend our test generation framework for dynamic behaviors to include nondeterministic behaviors. Nondeterminism often stems from probabilities and randomization, causing programs to branch into multiple outcomes despite identical inputs. We want to systematically explore as many of these outcomes as possible. Leveraging the insight from SynToken that there exist common patterns such as shared prefixes, we plan to define constraints to enable early termination when identical execution trace prefixes are detected, allowing us to improve testing efficiency while covering diverse execution paths.
Tue 24 JunDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
14:00 - 15:30 | Testing 3Research Papers / Industry Papers / Ideas, Visions and Reflections / Journal First at Cosmos 3C Chair(s): Dan Hao Peking University | ||
14:00 20mTalk | LlamaRestTest: Effective REST API Testing with Small Language Models Research Papers Myeongsoo Kim Georgia Institute of Technology, Saurabh Sinha IBM Research, Alessandro Orso University of Georgia, USA DOI | ||
14:20 20mTalk | Testing Updated Apps by Adapting Learned Models Journal First Chanh Duc Ngo University of Luxembourg, Fabrizio Pastore University of Luxembourg, Lionel Briand University of Ottawa, Canada; Lero centre, University of Limerick, Ireland Link to publication | ||
14:40 20mTalk | Automated Testing of COBOL to Java Transformation Industry Papers Sandeep Hans IBM India Research Lab, Atul Kumar IBM Research India, Toshiaki Yasue IBM Research - Tokyo, Kohichi Ono IBM Research - Tokyo, Saravanan Krishnan IBM India Research Lab, Devika Sondhi IBM Research, Fumiko Satoh IBM Research - Tokyo, Gerald Mitchell IBM Software, Sachin Kumar IBM Software, Diptikalyan Saha IBM Research India | ||
15:00 20mTalk | TerzoN: Human-in-the-Loop Software Testing with a Composite Oracle Research Papers Matthew C. Davis Carnegie Mellon University, Amy Wei University of Michigan, Brad A. Myers Carnegie Mellon University, Joshua Sunshine Carnegie Mellon University Link to publication DOI | ||
15:20 10mTalk | Efficient Test Generation for Dynamic Behaviors Leveraging Token-Level Input Commonalities Ideas, Visions and Reflections | ||
Cosmos 3C is the third room in the Cosmos 3 wing.
When facing the main Cosmos Hall, access to the Cosmos 3 wing is on the left, close to the stairs. The area is accessed through a large door with the number “3”, which will stay open during the event.