ICSE 2024
Fri 12 - Sun 21 April 2024 Lisbon, Portugal
Fri 19 Apr 2024 16:15 - 16:30 at Sophia de Mello Breyner Andresen - Testing of AI systems Chair(s): Aldeida Aleti

In the wake of developments in the field of Natural Language Processing, Question Answering (QA) software has penetrated our daily life. Due to the data-driven programming paradigm, QA software inevitably contains bugs, i.e., misbehaving in real-world applications. Current testing techniques for testing QA software include two folds, reference-based testing and metamorphic testing.

This paper adopts a different angle to achieve testing for QA software: we notice that answers to questions would have inference relations, i.e., the answers to some questions could be \textit{logically inferred} from the answers to other questions. If these answers on QA software do not satisfy the inference relations, an inference bug is detected. To generate the questions with the inference relations automatically, we propose a novel testing method \textbf{K}nowledge \textbf{G}raph driven \textbf{I}nference \textbf{T}esting (\textbf{KGIT}), which employs facts in the Knowledge Graph (KG) as the seeds to logically construct test cases containing questions and contexts with inference relations. To evaluate the effectiveness of KGIT, we conduct an extensive empirical study with more than 2.8 million test cases generated from the large-scale KG YAGO4 and three QA models based on the state-of-the-art QA model structure. The experimental results show that our method (a) could detect a considerable number of inference bugs in all three studied QA models and (b) is helpful in retraining QA models to improve their inference ability.

Fri 19 Apr

Displayed time zone: Lisbon change

16:00 - 17:30
Testing of AI systemsResearch Track / Journal-first Papers at Sophia de Mello Breyner Andresen
Chair(s): Aldeida Aleti Monash University
16:00
15m
Talk
CIT4DNN: Generating Diverse and Rare Inputs for Neural Networks Using Latent Space Combinatorial Testing
Research Track
Swaroopa Dola University of Virginia, Rory McDaniel University of Virginia, Matthew B Dwyer University of Virginia, Mary Lou Soffa University of Virginia
16:15
15m
Talk
Knowledge Graph Driven Inference Testing for Question Answering Software
Research Track
Jun Wang Nanjing University, Yanhui Li Nanjing University, Zhifei Chen Nanjing University, Lin Chen Nanjing University, Xiaofang Zhang Soochow University, Yuming Zhou Nanjing University
16:30
15m
Talk
DeepSample: DNN sampling-based testing for operational accuracy assessment
Research Track
Antonio Guerriero Università di Napoli Federico II, Roberto Pietrantuono Università di Napoli Federico II, Stefano Russo Università di Napoli Federico II
Pre-print
16:45
15m
Talk
MAFT: Efficient Model-Agnostic Fairness Testing for Deep Neural Networks via Zero-Order Gradient Search
Research Track
Zhaohui Wang East China Normal University, Min Zhang East China Normal University, Jingran Yang East China Normal University, ShaoBojie East China Normal University, Min Zhang East China Normal University
17:00
7m
Talk
DeepManeuver: Adversarial Test Generation for Trajectory Manipulation of Autonomous Vehicles
Journal-first Papers
Meriel von Stein University of Virginia, Sebastian Elbaum University of Virginia, David Shriver Software Engineering Institute
17:07
7m
Talk
Finding Deviated Behaviors of the Compressed DNN Models for Image Classifications
Journal-first Papers
Yongqiang Tian The Hong Kong University of Science and Technology; University of Waterloo, Wuqi Zhang The Hong Kong University of Science and Technology, Ming Wen Huazhong University of Science and Technology, Shing-Chi Cheung Hong Kong University of Science and Technology, Chengnian Sun University of Waterloo, Shiqing Ma UMass Amherst, Yu Jiang Tsinghua University
Link to publication DOI
17:14
7m
Talk
Identifying the Hazard Boundary of ML-enabled Autonomous Systems Using Cooperative Co-Evolutionary Search
Journal-first Papers
Sepehr Sharifi University of Ottawa, Donghwan Shin University of Sheffield, Lionel Briand University of Ottawa, Canada; Lero centre, University of Limerick, Ireland, Nathan Aschbacher Auxon Corporation
Hide past events