Metamorphic Testing of Machine Translation Models using Back Translation
Machine translation software has been widely adopted in recent years. The recent advance in deep learning research has massively improved the accuracy and fluency of the translated output. However, incorrect translations may still occur, which cause misunderstandings, and even more detrimental consequences when applying these systems for crucial applications, such as translating legal and medical documents. This calls for methods that can test the correctness of machine translation software efficiently and effectively. In this paper, we propose a method, which uses back-translation as a reference for machine translation testing, minimizing the knowledge and use of the NLP tools in the target language, so that the same workflow can be applied to test systems translating English to multiple languages. We build a metamorphic testing method using our proposed concept called contextual referentially transparent input (CRTI). A CRTI is a piece of text that should have a similar meaning under a certain context in any given language. Our method detects inconsistency between a CRTI in the original sentence and the back-translation to report translation errors. To evaluate our method, we translate 200 sentences using Google Translate. Our method reports 57 suspicious issues with a precision of 74% in Chinese translation and 22 suspicious issues with a precision of 82% in Vietnamese translation.
Mon 15 MayDisplayed time zone: Hobart change
13:45 - 15:15
|Metamorphic Testing of Machine Translation Models using Back Translation
|A Method of Identifying Causes of Prediction Errors to Accelerate MLOps
|DeepSHAP Summary for Adversarial Example Detection
|DeepPatch: A Patching-Based Method for Repairing Deep Neural Networks