Write a Blog >>
ICSE 2021
Mon 17 May - Sat 5 June 2021

Machine translation software has seen rapid progress in recent years due to the advancement of deep neural networks. People routinely use machine translation software in their daily lives, such as ordering food in a foreign restaurant, receiving medical diagnosis and treatment from foreign doctors, and reading international political news online. However, due to the complexity and intractability of the underlying neural networks, modern machine translation software is still far from robust and can produce poor or incorrect translations; this can lead to misunderstanding, financial loss, threats to personal safety and health, and political conflicts. To address this problem, we introduce referentially transparent inputs (RTIs), a simple, widely applicable methodology for validating machine translation software. A referentially transparent input is a piece of text that should have similar translations when used in different contexts. Our practical implementation, Purity, detects when this property is broken by a translation. To evaluate RTI, we use Purity to test Google Translate and Bing Microsoft Translator with 200 unlabeled sentences, which detected 123 and 142 erroneous translations with high precision (79.3% and 78.3%). The translation errors are diverse, including examples of under-translation, over-translation, word/phrase mistranslation, incorrect modification, and unclear logic.

Conference Day
Tue 25 May

Displayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change

12:05 - 13:05
1.2.1. Deep Neural Networks: Validation #2Technical Track at Blended Sessions Room 1 +12h
Chair(s): Grace LewisCarnegie Mellon Software Engineering Institute
12:05
20m
Paper
Measuring Discrimination to Boost Comparative Testing for Multiple Deep Learning ModelsTechnical Track
Technical Track
Linghan MengNanjing University, Yanhui LiDepartment of Computer Science and Technology, Nanjing University, Lin ChenDepartment of Computer Science and Technology, Nanjing University, Zhi WangNanjing University, Di WuMomenta, Yuming ZhouNanjing University, Baowen XuNanjing University
Pre-print Media Attached
12:25
20m
Paper
Prioritizing Test Inputs for Deep Neural Networks via Mutation AnalysisTechnical Track
Technical Track
Zan WangCollege of Intelligence and Computing, Tianjin University, Hanmo YouCollege of Intelligence and Computing, Tianjin University, Junjie ChenCollege of Intelligence and Computing, Tianjin University, Yingyi ZhangCollege of Intelligence and Computing, Tianjin University, Xuyuan DongInformation and Network Center,Tianjin University, Wenbin ZhangInformation and Network Center,Tianjin University
Pre-print Media Attached
12:45
20m
Paper
Testing Machine Translation via Referential TransparencyTechnical Track
Technical Track
Pinjia HeETH Zurich, Clara MeisterETH Zurich, Zhendong SuETH Zurich
Pre-print Media Attached

Conference Day
Wed 26 May

Displayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change

00:05 - 01:05
1.2.1. Deep Neural Networks: Validation #2Technical Track at Blended Sessions Room 1
00:05
20m
Paper
Measuring Discrimination to Boost Comparative Testing for Multiple Deep Learning ModelsTechnical Track
Technical Track
Linghan MengNanjing University, Yanhui LiDepartment of Computer Science and Technology, Nanjing University, Lin ChenDepartment of Computer Science and Technology, Nanjing University, Zhi WangNanjing University, Di WuMomenta, Yuming ZhouNanjing University, Baowen XuNanjing University
Pre-print Media Attached
00:25
20m
Paper
Prioritizing Test Inputs for Deep Neural Networks via Mutation AnalysisTechnical Track
Technical Track
Zan WangCollege of Intelligence and Computing, Tianjin University, Hanmo YouCollege of Intelligence and Computing, Tianjin University, Junjie ChenCollege of Intelligence and Computing, Tianjin University, Yingyi ZhangCollege of Intelligence and Computing, Tianjin University, Xuyuan DongInformation and Network Center,Tianjin University, Wenbin ZhangInformation and Network Center,Tianjin University
Pre-print Media Attached
00:45
20m
Paper
Testing Machine Translation via Referential TransparencyTechnical Track
Technical Track
Pinjia HeETH Zurich, Clara MeisterETH Zurich, Zhendong SuETH Zurich
Pre-print Media Attached