LEAP: Efficient and Automated Test Method for NLP SoftwareRecorded talk
The widespread adoption of DNNs in NLP software has highlighted the need for robustness. Researchers proposed various automatic testing techniques for adversarial test cases. However, existing methods suffer from two limitations: weak error-discovering capabilities, with success rates ranging from 0% to 24.6% for BERT-based NLP software, and time inefficiency, taking 177.8s to 205.28s per test case, making them challenging for time-constrained scenarios.
To address these issues, this paper proposes LEAP, an automated test method that uses LEvy flight-based Adaptive Particle swarm optimization integrated with textual features to generate adversarial test cases. Specifically, we adopt Levy flight for population initialization to increase the diversity of generated test cases. We also design an inertial weight adaptive update operator to improve the efficiency of LEAP’s global optimization of high-dimensional text examples and a mutation operator based on the greedy strategy to reduce the search time.
We conducted a series of experiments to validate LEAP’s ability to test NLP software and found that the average success rate of LEAP in generating adversarial test cases is 79.1%, which is 6.1% higher than the next best approach (PSOattack). While ensuring high success rates, LEAP significantly reduces time overhead by up to 147.6s compared to other heuristic-based methods. Additionally, the experimental results demonstrate that LEAP can generate more transferable test cases and significantly enhance the robustness of DNN-based systems.
Thu 14 SepDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
10:30 - 12:00 | Software Testing for Specialized Systems 2Research Papers / Tool Demonstrations at Room C Chair(s): Zishuo Ding University of Waterloo | ||
10:30 12mTalk | Bridging the Gap between Academia and Industry in Machine Learning Software Defect Prediction: Thirteen Considerations Research Papers Szymon Stradowski Nokia & Wrocław University of Science and Technology, Lech Madeyski Wroclaw University of Science and Technology Link to publication DOI Pre-print Media Attached | ||
10:42 12mTalk | Identify and Update Test Cases when Production Code Changes: A Transformer-based Approach Research Papers Xing Hu Zhejiang University, Zhuang Liu Zhejiang University, Xin Xia Huawei Technologies, Zhongxin Liu Zhejiang University, Tongtong Xu Huawei, Xiaohu Yang Zhejiang University | ||
10:54 12mTalk | Revisiting and Improving Retrieval-Augmented Deep Assertion Generation Research Papers Weifeng Sun , Hongyan Li Chongqing University, Meng Yan Chongqing University, Yan Lei Chongqing University, Hongyu Zhang Chongqing University, Hongyu Zhang Chongqing University | ||
11:06 12mTalk | Provengo: A Tool Suite for Scenario Driven Model-Based Testing Tool Demonstrations Michael Bar Sinai Provengo, Achiya Elyasaf Ben-Gurion University of the Negev, Gera Weiss Ben-Gurion University of the Negev, Yeshayahu Weiss Ben-Gurion University of the Negev Pre-print File Attached | ||
11:18 12mTalk | QuraTest: Integrating Quantum Specific Features in Quantum Program Testing Research Papers Jiaming Ye Kyushu University, Shangzhou Xia Kyushu University, Fuyuan Zhang Kyushu University, Paolo Arcaini National Institute of Informatics
, Lei Ma University of Alberta, Jianjun Zhao Kyushu University, Fuyuki Ishikawa National Institute of Informatics File Attached | ||
11:30 12mTalk | QuCAT: A Combinatorial Testing Tool for Quantum Software Tool Demonstrations Xinyi Wang Simula Research Laboratory, Paolo Arcaini National Institute of Informatics
, Tao Yue Beihang University, Shaukat Ali Simula Research Laboratory and Oslo Metropolitan University Pre-print File Attached | ||
11:42 12mTalk | LEAP: Efficient and Automated Test Method for NLP SoftwareRecorded talk Research Papers Mingxuan Xiao Hohai University, Yan Xiao National University of Singapore, Hai Dong RMIT University, Shunhui Ji Hohai University, Pengcheng Zhang Hohai University Media Attached | ||