SR-MT:A Metamorphic Method to Test Robustness of Speech Recognition Software
Speech recognition (SR) systems are systems that convert speech signals into text and are widely used in mobile devices, wearable devices, and intelligent living room devices. When SR software recognizes speech in the laboratory or another quiet environment, the recognition accuracy is high. However, when SR software is applied in a complex real-world environment, there may be background noise from birds, machines, and so on, the influence of other speakers, or other adverse factors, which will reduce the recognition accuracy of the SR software. In this work, we propose SR-MT, which is a metamorphic testing (MT) approach to test the robustness of SR software. In SR-MT, we adopt four criteria to analyze the MT results and rank the robustness of the target SR software for speech affected by noise interference and speech variations from level 1 to level 5. SR-MT was evaluated on three real industrial applications: iFLYTEK speech-to-text1, Baidu speech-to-text2, and Google speech-to-text3. We found that on average, 13.5% of the words in the speech could not be recognized correctly when the signal-to-noise ratio reached 10 dB. Similarly, changes in speech speed and tone will also reduce the recognition accuracy of SR software.
Mon 9 MayDisplayed time zone: Eastern Time (US & Canada) change
08:00 - 09:30 | Metamorphic Testing in Deep LearningMET at MET room Chair(s): Xiaoyuan Xie School of Computer Science, Wuhan University, China | ||
08:00 30mTalk | In-Place Metamorphic Exploration and Testing MET Zhi Quan (George) Zhou University of Wollongong, Australia, Junting Zhu University of Wollongong, Tsong Yueh Chen Swinburne University of Technology, Dave Towey University of Nottingham Ningbo China | ||
08:30 30mTalk | Fairness Evaluation in Deepfake Detection Models using Metamorphic Testing MET Muxin Pu Monash University Malaysia, Meng Yi Kuan Monash University Malaysia, Nyee Thoang Lim Monash University Malaysia, Mei Kuan Lim Monash University Malaysia, Chun Yong Chong Monash University Malaysia | ||
09:00 30mTalk | SR-MT:A Metamorphic Method to Test Robustness of Speech Recognition Software MET Feifei Wang Naval University of Engineering, Kerong Ben Naval University of Engineering, Xian Zhang Naval University of Engineering |