As automated speech recognition (ASR) systems gain widespread acceptance, there is a pressing need to rigorously test and enhance their performance. Nonetheless, the process of collecting and executing speech test cases is typically both costly and time-consuming. This presents a compelling case for the strategic prioritization of speech test cases, which consist of a piece of audio and the corresponding reference text. The central question we address is: In what sequence should speech test cases be collected and executed to identify the maximum number of errors at the earliest stage?
In this study, we introduce Prophet (PRiOritising sPeecH tEsT cases), a tool designed to predict the likelihood that speech test cases will identify errors. Consequently, Prophet can assess and prioritize these test cases without having to run the ASR system, facilitating large-scale analysis. Our evaluation encompasses 6 distinct prioritization techniques across 3 ASR systems and 12 datasets. When constrained by the same test budget, our approach identified 15.44% more misrecognized words than the leading the state-of-the-art method. We select top-ranked speech test cases from the prioritized list to fine-tune ASR systems and analyze how our approach can improve the ASR system performance. Statistical evaluations show that our method delivers a considerably higher performance boost for ASR systems compared to established baseline techniques. Moreover, our correlation analysis confirms that fine-tuning an ASR system with a dataset where the model initially underperforms tends to yield greater performance improvements. The implementation of Prophet, along with datasets and raw results, has been made available at https://github.com/yangzhou6666/ASRProphet.
The paper has been accepted for publication in ACM Transactions on Software Engineering and Methodology (TOSEM) on October 19, 2024, and is available at https://doi.org/10.1145/3707450.
Wed 25 JunDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
11:00 - 12:30 | Software TestsJournal First / Demonstrations / Research Papers at Pirsenteret 150 Chair(s): Tien N. Nguyen University of Texas at Dallas | ||
11:00 10mTalk | pytest-ranking: A Regression Test Prioritization Tool for Python Demonstrations Runxiang Cheng University of Illinois at Urbana-Champaign, Kaiyao Ke University of Illinois Urbana-Champaign, Darko Marinov University of Illinois at Urbana-Champaign | ||
11:10 20mTalk | The Effects of Computational Resources on Flaky Tests Journal First Denini Silva Federal University of Pernambuco, Martin Gruber BMW Group, Satyajit Gokhale Northeastern University, Ellen Arteca Northeastern University, Alexi Turcotte CISPA, Marcelo d'Amorim North Carolina State University, Wing Lam George Mason University, Stefan Winter LMU Munich, Jonathan Bell Northeastern University DOI File Attached | ||
11:30 20mTalk | Prioritizing Speech Test Cases Journal First Zhou Yang Singapore Management University; University of Alberta, Jieke Shi Singapore Management University, Muhammad Hilmi Asyrofi School of Computing and Information Systems, Singapore Management University, Bowen Xu North Carolina State University, Xin Zhou Singapore Management University, Singapore, DongGyun Han Royal Holloway, University of London, David Lo Singapore Management University | ||
11:50 20mTalk | Automated Unit Test Refactoring Research Papers Yi Gao Zhejiang University, Xing Hu Zhejiang University, Xiaohu Yang Zhejiang University, Xin Xia Zhejiang University DOI | ||
12:10 20mTalk | Understanding and Characterizing Mock Assertions in Unit Tests Research Papers Hengcheng Zhu The Hong Kong University of Science and Technology, Valerio Terragni University of Auckland, Lili Wei McGill University, Shing-Chi Cheung Hong Kong University of Science and Technology, Jiarong Wu , Yepang Liu Southern University of Science and Technology DOI Pre-print | ||
This room is located outside Clarion Hotel
This room is located in the Pirsenteret (The Pier Center) convention center. It is just outside the hotel, on the back, towards the fjord.
You should be able to go through the emergency exit at Clarion, just on the side of the Cosmos 3 wing, which will be bring you close to Pirsenteret.
The entrance to the center is from here:
https://maps.app.goo.gl/dU3qH6kAimXGBNHe7
Once inside, go all straight and you will find signage to reach the room. The room is known as room 150 inside the center.