Today’s software failures have two dominating root causes: code bugs and misconfigurations. To combat failure-inducing software changes, unified regression testing (URT) is needed to synergistically test the changed code and all changed production configurations (configs) for deployment reliability. However, URT could incur high cost to run a large number of tests under multiple configs. Regression test selection (RTS) can reduce regression testing cost. Unfortunately, no existing RTS technique reasons about code and config changes collectively. We introduce Unified Regression Testing Selection (uRTS) to effectively reduce the cost of URT. uRTS supports project changes on 1) code only, 2) configs only, and 3) both code and configs. It selects regular tests and configuration tests with a unified selection algorithm. The uRTS algorithm analyzes code and config dependencies of each test across runs and across configs. uRTS provides the same safety guarantee as the state-of-the-art RTS while selecting fewer tests and, more importantly, reducing the end-to-end testing time. We implemented uRTS on top of Ekstazi (a RTS tool for code changes) and Ctest (a config testing framework). We evaluate uRTS on hundreds of code revisions and dozens of configs of five large projects. The results show that uRTS reduces the end-to-end testing time, on average, by 3.64X compared to executing all tests and 1.87X compared to a competitive reference solution that directly extends RTS for URT.
Thu 18 MayDisplayed time zone: Hobart change
13:45 - 15:15 | Test quality and improvementTechnical Track / Journal-First Papers / DEMO - Demonstrations at Meeting Room 110 Chair(s): Guowei Yang University of Queensland | ||
13:45 15mTalk | Test Selection for Unified Regression Testing Technical Track Shuai Wang University of Illinois at Urbana-Champaign, Xinyu Lian University of Illinois at Urbana-Champaign, Darko Marinov University of Illinois at Urbana-Champaign, Tianyin Xu University of Illinois at Urbana-Champaign Pre-print | ||
14:00 15mTalk | ATM: Black-box Test Case Minimization based on Test Code Similarity and Evolutionary Search Technical Track Rongqi Pan University of Ottawa, Taher A Ghaleb University of Ottawa, Lionel Briand University of Luxembourg; University of Ottawa | ||
14:15 15mTalk | Measuring and Mitigating Gaps in Structural Testing Technical Track Soneya Binta Hossain University of Virginia, Matthew B Dwyer University of Virginia, Sebastian Elbaum University of Virginia, Anh Nguyen-Tuong University of Virginia Pre-print | ||
14:30 7mTalk | FlaPy: Mining Flaky Python Tests at Scale DEMO - Demonstrations Pre-print | ||
14:37 7mTalk | Scalable and Accurate Test Case Prioritization in Continuous Integration Contexts Journal-First Papers Ahmadreza Saboor Yaraghi University of Ottawa, Mojtaba Bagherzadeh University of Ottawa, Nafiseh Kahani University of Carlton, Lionel Briand University of Luxembourg; University of Ottawa | ||
14:45 7mTalk | Flakify: A Black-Box, Language Model-based Predictor for Flaky Tests Journal-First Papers Sakina Fatima University of Ottawa, Taher A Ghaleb University of Ottawa, Lionel Briand University of Luxembourg; University of Ottawa | ||
14:52 7mTalk | Developer-centric test amplification Journal-First Papers Pre-print | ||
15:00 7mTalk | How Developers Engineer Test Cases: An Observational Study Journal-First Papers Maurício Aniche Delft University of Technology, Christoph Treude University of Melbourne, Andy Zaidman Delft University of Technology Pre-print |