Automatic Fault Detection for Deep Learning Programs Using Graph Transformations
Wed 11 May 2022 11:05 - 11:10 at ICSE room 1-odd hours - Machine Learning with and for SE 10 Chair(s): Preetha Chatterjee
Nowadays, we are witnessing an increasing demand in both corporates and academia for exploiting Deep Learning (DL) to solve complex real-world problems. A DL program encodes the network structure of a desirable DL model and the process by which the model learns from the training dataset. Like any software, a DL program can be faulty, which implies substantial challenges of software quality assurance, especially in safety-critical domains. It is therefore crucial to equip DL development teams with efficient fault detection techniques and tools. In this article, we propose NeuraLint, a model-based fault detection approach for DL programs, using meta-modeling and graph transformations. First, we design a meta-model for DL programs that includes their base skeleton and fundamental properties. Then, we construct a graph-based verification process that covers 23 rules defined on top of the meta-model and implemented as graph transformations to detect faults and design inefficiencies in the generated models (i.e., instances of the meta-model). First, the proposed approach is evaluated by finding faults and design inefficiencies in 28 synthesized examples built from common problems reported in the literature. Then NeuraLint successfully finds 64 faults and design inefficiencies in 34 real-world DL programs extracted from Stack Overflow posts and GitHub repositories. The results show that NeuraLint effectively detects faults and design issues in both synthesized and real-world examples with a recall of 70.5% and a precision of 100%. Although the proposed meta-model is designed for feedforward neural networks, it can be extended to support other neural network architectures such as recurrent neural networks. Researchers can also expand our set of verification rules to cover more types of issues in DL programs.
Mon 9 MayDisplayed time zone: Eastern Time (US & Canada) change
Wed 11 MayDisplayed time zone: Eastern Time (US & Canada) change
11:00 - 12:00 | Machine Learning with and for SE 10Technical Track / SEIP - Software Engineering in Practice / Journal-First Papers at ICSE room 1-odd hours Chair(s): Preetha Chatterjee Drexel University, USA | ||
11:00 5mTalk | Defect Reduction Planning (using TimeLIME) Journal-First Papers Authorizer link Pre-print Media Attached | ||
11:05 5mTalk | Automatic Fault Detection for Deep Learning Programs Using Graph Transformations Journal-First Papers Amin Nikanjam École Polytechnique de Montréal, Houssem Ben Braiek École Polytechnique de Montréal, Mohammad Mehdi Morovati École Polytechnique de Montréal, Foutse Khomh Polytechnique Montréal Link to publication DOI Media Attached | ||
11:10 5mTalk | Counterfactual Explanations for Models of Code SEIP - Software Engineering in Practice Jürgen Cito TU Wien and Meta, Işıl Dillig University of Texas at Austin, Vijayaraghavan Murali Meta Platforms, Inc., Satish Chandra Facebook Pre-print Media Attached | ||
11:15 5mTalk | VarCLR: Variable Semantic Representation Pre-training via Contrastive Learning Technical Track Qibin Chen Carnegie Mellon University, Jeremy Lacomis Carnegie Mellon University, Edward J. Schwartz Carnegie Mellon University Software Engineering Institute, Graham Neubig Carnegie Mellon University, Bogdan Vasilescu Carnegie Mellon University, USA, Claire Le Goues Carnegie Mellon University DOI Pre-print Media Attached | ||
11:20 5mTalk | Towards Training Reproducible Deep Learning Models Technical Track Boyuan Chen Centre for Software Excellence, Huawei Canada, Mingzhi Wen Huawei Technologies, Yong Shi Huawei Technologies, Dayi Lin Centre for Software Excellence, Huawei, Canada, Gopi Krishnan Rajbahadur Centre for Software Excellence, Huawei, Canada, Zhen Ming (Jack) Jiang York University Pre-print Media Attached | ||
11:25 5mTalk | Learning to Reduce False Positives in Analytic Bug Detectors Technical Track Anant Kharkar Microsoft, Roshanak Zilouchian Moghaddam Microsoft, Matthew Jin Microsoft Corporation, Xiaoyu Liu Microsoft Corporation, Xin Shi Microsoft Corporation, Colin Clement Microsoft, Neel Sundaresan Microsoft Corporation Pre-print Media Attached |