CIT4DNN: Generating Diverse and Rare Inputs for Neural Networks Using Latent Space Combinatorial Testing
Deep neural networks (DNN) are being used in a wide range of applications including safety-critical systems. Several DNN test generation approaches have been proposed to generate fault-revealing test inputs. However, the existing test generation approaches do not systematically cover the input data distribution to test DNNs with diverse inputs, and none of the approaches investigate the relationship between rare inputs and faults. We propose CIT4DNN, an automated black-box approach to generate DNN test sets that are feature-diverse and fault-revealing. CIT4DNN constructs diverse test sets by applying combinatorial interaction testing to the latent space of generative models and formulates constraints over the geometry of the latent space to generate rare and fault-revealing test inputs. Evaluation on a range of datasets and models shows that CIT4DNN generated tests are more feature diverse than the state-of-the-art, and can target rare fault-revealing testing inputs more effectively than existing methods.
Fri 19 AprDisplayed time zone: Lisbon change
16:00 - 17:30 | Testing of AI systemsResearch Track / Journal-first Papers at Sophia de Mello Breyner Andresen Chair(s): Aldeida Aleti Monash University | ||
16:00 15mTalk | CIT4DNN: Generating Diverse and Rare Inputs for Neural Networks Using Latent Space Combinatorial Testing Research Track Swaroopa Dola University of Virginia, Rory McDaniel University of Virginia, Matthew B Dwyer University of Virginia, Mary Lou Soffa University of Virginia | ||
16:15 15mTalk | Knowledge Graph Driven Inference Testing for Question Answering Software Research Track Jun Wang Nanjing University, Yanhui Li Nanjing University, Zhifei Chen Nanjing University, Lin Chen Nanjing University, Xiaofang Zhang Soochow University, Yuming Zhou Nanjing University | ||
16:30 15mTalk | DeepSample: DNN sampling-based testing for operational accuracy assessment Research Track Antonio Guerriero Università di Napoli Federico II, Roberto Pietrantuono Università di Napoli Federico II, Stefano Russo Università di Napoli Federico II Pre-print | ||
16:45 15mTalk | MAFT: Efficient Model-Agnostic Fairness Testing for Deep Neural Networks via Zero-Order Gradient Search Research Track Zhaohui Wang East China Normal University, Min Zhang East China Normal University, Jingran Yang East China Normal University, ShaoBojie East China Normal University, Min Zhang East China Normal University | ||
17:00 7mTalk | DeepManeuver: Adversarial Test Generation for Trajectory Manipulation of Autonomous Vehicles Journal-first Papers Meriel von Stein University of Virginia, Sebastian Elbaum University of Virginia, David Shriver Software Engineering Institute | ||
17:07 7mTalk | Finding Deviated Behaviors of the Compressed DNN Models for Image Classifications Journal-first Papers Yongqiang Tian The Hong Kong University of Science and Technology; University of Waterloo, Wuqi Zhang The Hong Kong University of Science and Technology, Ming Wen Huazhong University of Science and Technology, Shing-Chi Cheung Hong Kong University of Science and Technology, Chengnian Sun University of Waterloo, Shiqing Ma University of Massachusetts, Amherst, Yu Jiang Tsinghua University Link to publication DOI | ||
17:14 7mTalk | Identifying the Hazard Boundary of ML-enabled Autonomous Systems Using Cooperative Co-Evolutionary Search Journal-first Papers Sepehr Sharifi University of Ottawa, Donghwan Shin University of Sheffield, Lionel Briand University of Ottawa, Canada; Lero centre, University of Limerick, Ireland, Nathan Aschbacher Auxon Corporation | ||