Write a Blog >>
ICSE 2020
Wed 24 June - Thu 16 July 2020
Sat 11 Jul 2020 00:12 - 00:24 at Goguryeo - P26-Deep Learning Testing and Debugging Chair(s): Tim Menzies

In recent years, machine translation software has increasingly been integrated into our daily lives. People routinely use machine translation for various applications, such as describing symptoms to a foreign doctor and reading political news in a foreign language. However, due to the complexity and intractability of neural machine translation (NMT) models that power modern machine translation systems, these systems are far from being robust. They can return inferior results that lead to misunderstanding, medical misdiagnoses, threats to personal safety, or political conflicts. Despite its apparent importance, validating the robustness of machine translation is very difficult and has, therefore, been much under-explored.

To tackle this challenge, we introduce \textit{structure-invariant testing (SIT)}, a novel metamorphic testing approach for validating machine translation software. Our key insight is that the translation results of “similar” source sentences should typically exhibit a similar sentence structure. Specifically, SIT (1) generates similar source sentences by substituting one word in a given sentence with semantically similar, syntactically equivalent words, and (2) represents sentence structure by syntax parse trees (obtained via constituency or dependency parsing). To evaluate SIT, we have used it to test Google Translate and Bing Microsoft Translator with 200 source sentences as input, which led to 64 and 70 buggy translations with 69.5% and 70% top-1 accuracy, respectively. The bugs are diverse, including under-translation, over-translation, incorrect modification, word/phrase mistranslation, and unclear logic, none of which could be detected via existing translation quality metrics.

Sat 11 Jul

Displayed time zone: (UTC) Coordinated Universal Time change

00:00 - 01:00
P26-Deep Learning Testing and DebuggingTechnical Papers at Goguryeo
Chair(s): Tim Menzies North Carolina State University
00:00
12m
Talk
ReluDiff: Differential Verification of Deep Neural NetworksArtifact ReusableTechnical
Technical Papers
Brandon Paulsen University of Southern California, Jingbo Wang University of Southern California, Chao Wang USC
Pre-print
00:12
12m
Talk
Structure-Invariant Testing for Machine TranslationTechnical
Technical Papers
Pinjia He ETH Zurich, Clara Meister ETH Zurich, Zhendong Su ETH Zurich, Switzerland
00:24
12m
Talk
Automatic Testing and Improvement of Machine TranslationTechnical
Technical Papers
Zeyu Sun Peking University, Jie M. Zhang University College London, UK, Mark Harman Facebook and University College London, Mike Papadakis University of Luxembourg, Lu Zhang Peking University, China
00:36
12m
Talk
Testing DNN Image Classifier for Confusion & Bias ErrorsArtifact ReusableTechnicalArtifact Available
Technical Papers
Yuchi Tian Columbia University, Ziyuan Zhong Columbia University, Vicente Ordonez University of Virginia, Gail Kaiser Columbia University, Baishakhi Ray Columbia University, New York
00:48
12m
Talk
Repairing Deep Neural Networks: Fix Patterns and ChallengesArtifact ReusableTechnicalArtifact Available
Technical Papers
Md Johirul Islam Iowa State University, Rangeet Pan Iowa State University, USA, Giang Nguyen Dept. of Computer Science, Iowa State University, Hridesh Rajan Iowa State University, USA