Write a Blog >>
Tue 11 Oct 2022 10:30 - 10:50 at Ballroom C East - Technical Session 1 - AI for SE I Chair(s): Andrea Stocco

AI-enabled software systems (AIS) are prevalent in a wide range of applications, such as visual tasks of autonomous systems, extensively deployed in automotive, aerial, and naval domains. Hence, it is crucial for human to evaluate the model’s intelligence before AIS is deployed to safety-critical environments, such as public roads.

In this paper, we assess AIS visual intelligence through measuring the completeness of its perception of primary concepts in a domain and the concept variants. For instance, is the visual perception of an autonomous detector mature enough to recognize the instances of \textit{pedestrian} (an automotive domain’s concept) in Halloween customs? An AIS will be more reliable once the model’s ability to perceive a concept is displayed in a human-understandable language. For instance, is the pedestrian in \textit{wheelchair} mistakenly recognized as a pedestrian on \textit{bike}, since the domain concepts bike and wheelchair, both associate with a mutual feature \textit{wheel}?

We answer the above-type questions by implementing a generic process within a framework, called B-AIS, which systematically evaluates AIS perception against the semantic specifications of a domain, while treating the model as a black-box. Semantics is the meaning and understanding of words in a language, and therefore, is more comprehensible for human brain than AIS pixel-level visual information. B-AIS processes the heterogeneous artifacts to be comparable, and leverages the comparison’s results to reveal AIS weaknesses in a human-understandable language. The evaluations of B-AIS for the vision task of pedestrian detection showed B-AIS identified the missing variants of the pedestrian with $F_{2}$ measures of 95% and in the dataset and 85% in the model.

Tue 11 Oct

Displayed time zone: Eastern Time (US & Canada) change

10:30 - 12:30
Technical Session 1 - AI for SE IResearch Papers / Industry Showcase at Ballroom C East
Chair(s): Andrea Stocco Università della Svizzera italiana (USI)
10:30
20m
Research paper
B-AIS: An Automated Process for Black-box Evaluation of AI-enabled Software Systems against Domain Semantics
Research Papers
Hamed Barzamini , Mona Rahimi Northern Illinois University
10:50
20m
Industry talk
Automatic Generation of Visualizations for Machine Learning Pipelines
Industry Showcase
Lei Liu Fujitsu Laboratories of America, Inc., Wei-Peng Chen Fujitsu Research of America, Inc., Mehdi Bahrami Fujitsu Laboratories of America, Inc., Mukul Prasad Amazon Web Services
11:10
20m
Research paper
SmOOD: Smoothness-based Out-of-Distribution Detection Approach for Surrogate Neural Networks in Aircraft DesignVirtual
Research Papers
Houssem Ben Braiek École Polytechnique de Montréal, Ali Tfaily Bombardier Aerospace, Foutse Khomh Polytechnique Montréal, Thomas Reid , Ciro Guida Bombardier Aerospace
Pre-print
11:30
20m
Research paper
Unveiling Hidden DNN Defects with Decision-Based Metamorphic TestingVirtual
Research Papers
Yuanyuan Yuan The Hong Kong University of Science and Technology, Qi Pang HKUST, Shuai Wang Hong Kong University of Science and Technology
11:50
20m
Research paper
Patching Weak Convolutional Neural Network Models through Modularization and CompositionVirtual
Research Papers
Binhang Qi Beihang University, Hailong Sun Beihang University, Xiang Gao Beihang University, China, Hongyu Zhang University of Newcastle
12:10
20m
Research paper
Safety and Performance, Why not Both? Bi-Objective Optimized Model Compression toward AI Software DeploymentVirtual
Research Papers
Jie Zhu Peking University, Leye Wang Peking University, Xiao Han Shanghai University of Finance and Economics
DOI Pre-print