T-Evos: A Large-Scale Longitudinal Study on CI Test Execution and Failure
Continuous integration is widely adopted in software projects to reduce the time it takes to deliver the changes to the market. To ensure software quality, developers also run regression test cases in a continuous fashion. The CI practice generates commit-by-commit software evolution data that provides great opportunities for future testing research. However, such data is often unavailable due to space limitation (e.g., developers only keep the data for a certain period) and the significant effort involved in rerunning the test cases on a per-commit basis. In this paper, we present T-Evos, a dataset on test result and coverage evolution, covering 8,093 commits across 12 open-source Java projects. Our dataset includes the evolution of statement-level code coverage for every test case (either passed and failed), test result, all the builds information, code changes, and the corresponding bug reports. We conduct an initial analysis to demonstrate the overall dataset. In addition, we conduct an empirical study using T-Evos to study the characteristics of test failures in CI settings. We find that test failures are frequent, and while most failures are resolved within a day, some failures require several weeks to resolve. We highlight the relationship between code changes and test failure, and provide insights for future automated testing research. Our dataset may be used for future testing research and benchmarking in CI. Our findings provide an important first step in understanding code coverage evolution and test failures in a continuous environment.
Wed 13 SepDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
15:30 - 17:00 | Software Testing for Specialized Systems 1Research Papers / Journal-first Papers / NIER Track at Room C Chair(s): Fabrizio Pastore University of Luxembourg | ||
15:30 12mTalk | DCLink: Bridging Data Constraint Changes and Implementations in FinTech Systems Research Papers Wensheng Tang Hong Kong University of Science and Technology, Chengpeng Wang Hong Kong University of Science and Technology, Peisen Yao Zhejing University, Rongxin Wu Xiamen University, Xianjin Fu Ant Group, Gang Fan Ant Group, Charles Zhang Hong Kong University of Science and Technology File Attached | ||
15:42 12mTalk | Systematically Detecting Packet Validation Vulnerabilities in Embedded Network Stacks Research Papers Paschal Amusuo Purdue University, Ricardo Andrés Calvo Méndez Universidad Nacional de Colombia, Zhongwei Xu Xi'an JiaoTong University, Aravind Machiry Purdue University, James C. Davis Purdue University Pre-print Media Attached File Attached | ||
15:54 12mTalk | WADIFF: A Differential Testing Framework for WebAssembly Runtimes Research Papers Shiyao Zhou The Hong Kong Polytechnic University, Muhui Jiang The Hong Kong Polytechnic University, Weimin Chen The Hong Kong Polytechnic University, Hao Zhou Hong Kong Polytechnic University, Haoyu Wang Huazhong University of Science and Technology, Xiapu Luo Hong Kong Polytechnic University File Attached | ||
16:06 12mTalk | T-Evos: A Large-Scale Longitudinal Study on CI Test Execution and Failure Journal-first Papers An Ran Chen University of Alberta, Tse-Hsun (Peter) Chen Concordia University, Shaowei Wang University of Manitoba Pre-print | ||
16:18 12mTalk | VRGuide: Efficient Testing of Virtual Reality Scenes via Dynamic Cut Coverage Research Papers Xiaoyin Wang University of Texas at San Antonio, Tahmid Rafi University of Texas at San Antonio, Na Meng Virginia Tech File Attached | ||
16:30 12mTalk | PURLTL: Mining LTL Specification from Imperfect Traces in TestingRecorded talk NIER Track Bo Peng Sun Yat-Sen University, Pingjia Liang Sun Yat-Sen University, Tingchen Han Sun Yat-Sen University, Weilin Luo Sun Yat-Sen University, Jianfeng Du Guangdong University of Foreign Studies, Hai Wan School of Data and Computer Science, Sun Yat-sen University, Rongzhen Ye Sun Yat-Sen University, Yuhang Zheng Sun Yat-Sen University Media Attached |