Fairness Evaluation in Deepfake Detection Models using Metamorphic Testing
Fairness of deepfake detectors in the presence of anomalies are not well investigated, especially if those anomalies are more prominent in either male or female subjects. The primary motivation for this work is to evaluate how deepfake detection model behaves under such anomalies. However, due to the black-box nature of deep learning (DL) and artificial intelligence (AI) systems, it is hard to predict the performance of a model when the input data is modified. Crucially, if this defect is not addressed properly, it will adversely affect the fairness of the model and result in discrimination of certain sub-population unintentionally. Therefore, the objective of this work is to adopt metamorphic testing to examine the reliability of the selected deepfake detection model, and how the transformation of input variation places influence on the output. We have chosen MesoInception-4, a state-of-the-art deepfake detection model, as the target model and makeup as the anomalies.
Mon 9 MayDisplayed time zone: Eastern Time (US & Canada) change
08:00 - 09:30 | Metamorphic Testing in Deep LearningMET at MET room Chair(s): Xiaoyuan Xie School of Computer Science, Wuhan University, China | ||
08:00 30mTalk | In-Place Metamorphic Exploration and Testing MET Zhi Quan (George) Zhou University of Wollongong, Australia, Junting Zhu University of Wollongong, Tsong Yueh Chen Swinburne University of Technology, Dave Towey University of Nottingham Ningbo China | ||
08:30 30mTalk | Fairness Evaluation in Deepfake Detection Models using Metamorphic Testing MET Muxin Pu Monash University Malaysia, Meng Yi Kuan Monash University Malaysia, Nyee Thoang Lim Monash University Malaysia, Mei Kuan Lim Monash University Malaysia, Chun Yong Chong Monash University Malaysia | ||
09:00 30mTalk | SR-MT:A Metamorphic Method to Test Robustness of Speech Recognition Software MET Feifei Wang Naval University of Engineering, Kerong Ben Naval University of Engineering, Xian Zhang Naval University of Engineering |