Prioritization of Test Cases with Varying Test Costs and Fault Severities for Certification Testing
We present an industrial case study on the application of test case prioritization techniques in the context of certification testing in consumer electronics domain. Test execution times and fault severities are subject to high variations in this domain. As a result, most of the existing techniques and metrics turn out to be inappropriate for this application context. We discuss such deficiencies and the room for improvement based on our case study with the certification test suites of 3 Smart TV applications as real experimental objects. We also propose a new metric, LAPFD, which is based on the calculation of the average of the percentage of faults detected. This calculation is weighted according to the cost of test cases and calculated separately per severity class. Then, a lexicographic ordering is performed based on these classes. We compared the baseline (random) ordering of test cases with respect to an alternative ordering based on cost, measured as the test execution time. These alternative orderings are evaluated by using the LAPFD metric. We observed that cost-based ordering of test cases consistently outperformed random ordering. Another observation is that there is a large room for improvement regarding the effectiveness of test case prioritization in this application domain.
Sat 24 OctDisplayed time zone: Lisbon change
14:20 - 15:55 | |||
14:20 15mTalk | On Using k-means Clustering for Test Suite Reduction TAIC PART 2020 Link to publication DOI | ||
14:35 15mTalk | Prioritization of Test Cases with Varying Test Costs and Fault Severities for Certification Testing TAIC PART 2020 Link to publication DOI | ||
14:50 15mTalk | Test Tools: an illusion of usability? TAIC PART 2020 Link to publication DOI | ||
15:05 15mTalk | Towards a unified catalog of attributes to guide industry in software testing technique selection TAIC PART 2020 Italo Santos University of São Paulo, Silvana Morita Melo , Paulo Sergio Lopes de Souza University of São Paulo - USP, Simone do Rocio Senger de Souza University of São Paulo - USP Link to publication DOI | ||
15:20 15mTalk | Data Dynamics for Testing Systems Fast Abstract: Data Dynamics for Testing Systems TAIC PART 2020 Julian Harty Commercetest Ltd Link to publication DOI | ||
15:35 20mLive Q&A | Q&A TAIC PART 2020 |