QATest: A Uniform Fuzzing Framework for Question Answering Systems
VirtualACM SIGSOFT Distinguished Paper Award
The tremendous advancements in deep learning techniques have empowered question answering(QA) systems with the capability of dealing with various tasks. Many commercial QA systems, such as Siri, Google Home, and Alexa, have been deployed to assist people in different daily activities. However, modern QA systems are often designed to deal with different topics and task formats, which makes both the test collection and labeling tasks difficult and thus threats their quality.
To alleviate this challenge, in this paper, we design and implement a fuzzing framework for QA systems, namely QATest, based on the metamorphic testing theory. It provides the first uniform solution to generate tests with oracle information automatically for various QA systems, such as machine reading comprehension, open-domain QA, and QA on knowledge bases. To further improve testing efficiency and generate more tests detecting erroneous behaviors, we design N-gram coverage criteria and perplexity priority based on the features of the question data to guide the generation process. To evaluate the performance of QATest, we experiment it on four QA systems that are designed for different tasks. The experiment results show that the tests generated by QATest detect hundreds of erroneous behaviors of QA systems efficiently. Also, the results confirm that the testing criteria can improve test diversity and fuzzing efficiency.
Tue 11 OctDisplayed time zone: Eastern Time (US & Canada) change
14:00 - 15:30
Technical Session 7 - Fuzzing IIResearch Papers at Banquet B
Chair(s): Karine Even-Mendoza Imperial College London
|So Many Fuzzers, So Little Time - Experience from Evaluating Fuzzers on the Contiki-NG Network (Hay)StackVirtual|
Clement Poncelet Uppsala University, Konstantinos (Kostis) Sagonas Uppsala University and Nat. Tech. Univ. of Athens, Nicolas Tsiftes RISE Research Institutes of SwedenDOI Pre-print
|FuzzerAid: Grouping Fuzzed Crashes Based On Fault Signatures|
Ashwin Kallingal Joshy Iowa State University, Wei Le Iowa State University
|QATest: A Uniform Fuzzing Framework for Question Answering SystemsVirtualACM SIGSOFT Distinguished Paper Award|
Zixi Liu Nanjing University, Yang Feng Nanjing University, Yining Yin Nanjing University, China, Jingyu Sun Nanjing University, Zhenyu Chen Nanjing University, Baowen Xu Nanjing University
|Effectively Generating Vulnerable Transaction Sequences in Smart Contracts with Reinforcement Learning-guided FuzzingVirtual|
Jianzhong Su Sun Yat-sen University, Hong-Ning Dai Hong Kong Baptist University, Lingjun Zhao Sun Yat-sen University, Zibin Zheng School of Data and Computer Science, Sun Yat-sen University, Xiapu Luo Hong Kong Polytechnic University