MAFT: Efficient Model-Agnostic Fairness Testing for Deep Neural Networks via Zero-Order Gradient Search
Deep neural networks (DNNs) have shown powerful performance in various applications and are increasingly being used in decision-making systems. However, concerns about fairness in DNNs always persist. Some efficient white-box fairness testing methods about individual fairness have been proposed. However, the development of black-box methods has stagnated, and the performance of existing methods is far behind that of white-box methods. In this paper, we propose a novel black-box individual fairness testing method called Model-Agnostic Fairness Testing (MAFT). By leveraging MAFT, practitioners can effectively identify and address discrimination in DL models, regardless of the specific algorithm or architecture employed. Our approach adopts lightweight procedures such as gradient es timation and attribute perturbation rather than trival procedures like symbol execution, rendering it significantly more scalable and applicable than existing methods. We demonstrate that MAFT achieves the same effectiveness as state-of-the-art white-box methods whilst improving the applicability to large-scale networks. Compared to existing black-box approaches, our approach demonstrates distinguished performance in discovering fairness violations w.r.t effectiveness (∼ 14.69×) and efficiency (∼ 32.58×).
Fri 19 AprDisplayed time zone: Lisbon change
16:00 - 17:30 | Testing of AI systemsResearch Track / Journal-first Papers at Sophia de Mello Breyner Andresen Chair(s): Aldeida Aleti Monash University | ||
16:00 15mTalk | CIT4DNN: Generating Diverse and Rare Inputs for Neural Networks Using Latent Space Combinatorial Testing Research Track Swaroopa Dola University of Virginia, Rory McDaniel University of Virginia, Matthew B Dwyer University of Virginia, Mary Lou Soffa University of Virginia | ||
16:15 15mTalk | Knowledge Graph Driven Inference Testing for Question Answering Software Research Track Jun Wang Nanjing University, Yanhui Li Nanjing University, Zhifei Chen Nanjing University, Lin Chen Nanjing University, Xiaofang Zhang Soochow University, Yuming Zhou Nanjing University | ||
16:30 15mTalk | DeepSample: DNN sampling-based testing for operational accuracy assessment Research Track Antonio Guerriero Università di Napoli Federico II, Roberto Pietrantuono Università di Napoli Federico II, Stefano Russo Università di Napoli Federico II Pre-print | ||
16:45 15mTalk | MAFT: Efficient Model-Agnostic Fairness Testing for Deep Neural Networks via Zero-Order Gradient Search Research Track Zhaohui Wang East China Normal University, Min Zhang East China Normal University, Jingran Yang East China Normal University, ShaoBojie East China Normal University, Min Zhang East China Normal University | ||
17:00 7mTalk | DeepManeuver: Adversarial Test Generation for Trajectory Manipulation of Autonomous Vehicles Journal-first Papers Meriel von Stein University of Virginia, Sebastian Elbaum University of Virginia, David Shriver Software Engineering Institute | ||
17:07 7mTalk | Finding Deviated Behaviors of the Compressed DNN Models for Image Classifications Journal-first Papers Yongqiang Tian The Hong Kong University of Science and Technology; University of Waterloo, Wuqi Zhang The Hong Kong University of Science and Technology, Ming Wen Huazhong University of Science and Technology, Shing-Chi Cheung Hong Kong University of Science and Technology, Chengnian Sun University of Waterloo, Shiqing Ma University of Massachusetts, Amherst, Yu Jiang Tsinghua University Link to publication DOI | ||
17:14 7mTalk | Identifying the Hazard Boundary of ML-enabled Autonomous Systems Using Cooperative Co-Evolutionary Search Journal-first Papers Sepehr Sharifi University of Ottawa, Donghwan Shin University of Sheffield, Lionel Briand University of Ottawa, Canada; Lero centre, University of Limerick, Ireland, Nathan Aschbacher Auxon Corporation |