Evaluating the Trade-offs of Text-based Diversity in Test Prioritization
Diversity-based techniques (DBT) have been cost-effective by prioritizing the most dissimilar test cases to detect faults at earlier stages of test execution. Diversity is measured on test specifications to convey how different test cases are from one another. However, there is little research on the trade-off of diversity measures based on different types of text-based specification (lexicographical or semantics). Particularly because the text content in test scripts vary widely from unit (e.g., code) to system-level (e.g., natural language). This paper compares and evaluates the cost-effectiveness in coverage and failures of different text-based diversity measures for different levels of tests. We perform an experiment on the test suites of 7 open source projects on the unit level, and 2 industry projects on the integration and system level. Our results show that test suites prioritised using semantic-based diversity measures causes a small improvement in requirements coverage, as opposed to lexical diversity that performed worse than random for system-level artefacts. In contrast, using lexical-based measures such as Jaccard or Levenshtein to prioritise code artefacts yield better failure coverage across all levels of tests. We summarise our findings into a list of recommendations for using semantic or lexical diversity on different levels of testing.
Tue 16 MayDisplayed time zone: Hobart change
13:45 - 15:15 | |||
13:45 22mTalk | Orchestration Strategies for Regression Test Suites AST 2023 Renan Greca Gran Sasso Science Institute, ISTI-CNR, Breno Miranda Federal University of Pernambuco, Antonia Bertolino National Research Council, Italy Pre-print | ||
14:07 22mTalk | Evaluating the Trade-offs of Text-based Diversity in Test Prioritization AST 2023 Ranim Khojah Chalmers | University of Gothenburg, Chi Hong Chao Chalmers | University of Gothenburg, Francisco Gomes de Oliveira Neto Chalmers University of Technology, Sweden / University of Gothenburg, Sweden | ||
14:30 22mTalk | MuTCR: Test Case Recommendation via Multi-Level Signature Matching AST 2023 Weisong Sun Nanjing University, Weidong Qian China Ship Scientific Research Center, Bin Luo Nanjing University, Zhenyu Chen Nanjing University | ||
14:52 22mTalk | Test Case Prioritization using Transfer Learning in Continuous Integration Environments AST 2023 Rezwana Mamata Ontario Tech University, Akramul Azim Ontario Tech University, Ramiro Liscano Ontario Tech University, Kevin Smith International Business Machines Corporation (IBM), Yee-Kang Chang International Business Machines Corporation (IBM), Gkerta Seferi International Business Machines Corporation (IBM), Qasim Tauseef International Business Machines Corporation (IBM) |