Empirically Revisiting and Enhancing IR-Based Test-Case Prioritization
Test-case prioritization (TCP) aims to detect regression bugs faster via reordering the tests run. While TCP has been studied for over 20 years, it was never evaluated with real test failures from open-source projects. In this work, we study the recent change-aware information retrieval (IR) technique for TCP. Prior work has shown it performing better than traditional coverage-based TCP techniques, but it was only evaluated on a small-scale dataset with a cost-unaware metric based on seeded faults/mutants. We extend the prior work by conducting a much larger and more realistic evaluation as well as proposing enhancements that substantially improve the performance. In particular, we evaluate the original technique on a large-scale, real-world software-evolution dataset with real failures using both cost-aware and cost-unaware metrics under various configurations. Also, we design and evaluate hybrid techniques combining the IR features, historical test execution time, and test failure frequencies. Our results show that the change-aware IR TCP technique outperforms state-of-the-art coverage-based techniques in this real-world setting, and our hybrid techniques improve even further upon the original IR technique. As a result, we explore various configurations of Moreover, we show that flaky tests have a substantial impact on evaluating the change-aware TCP techniques based on real test failures.
Tue 21 JulDisplayed time zone: Tijuana, Baja California change
14:50 - 15:50 | REGRESSION TESTINGTechnical Papers at Zoom Chair(s): Alex Orso Georgia Institute of Technology Public Live Stream/Recording. Registered participants should join via the Zoom link distributed in Slack. | ||
14:50 20mTalk | Dependent-Test-Aware Regression Testing Techniques Technical Papers Wing Lam University of Illinois at Urbana-Champaign, August Shi The University of Texas at Austin, Reed Oei , Sai Zhang Google Cloud, Michael D. Ernst University of Washington, USA, Tao Xie Peking University DOI Media Attached | ||
15:10 20mTalk | Differential Regression Testing for REST APIs Technical Papers Patrice Godefroid Microsoft Research, Daniel Lehmann University of Stuttgart, Marina Polishchuk Microsoft DOI Media Attached | ||
15:30 20mTalk | Empirically Revisiting and Enhancing IR-Based Test-Case Prioritization Technical Papers Qianyang Peng , August Shi The University of Texas at Austin, Lingming Zhang The University of Texas at Dallas DOI |