ASE 2023
Mon 11 - Fri 15 September 2023 Kirchberg, Luxembourg
Wed 13 Sep 2023 16:06 - 16:18 at Room C - Software Testing for Specialized Systems 1 Chair(s): Fabrizio Pastore

Continuous integration is widely adopted in software projects to reduce the time it takes to deliver the changes to the market. To ensure software quality, developers also run regression test cases in a continuous fashion. The CI practice generates commit-by-commit software evolution data that provides great opportunities for future testing research. However, such data is often unavailable due to space limitation (e.g., developers only keep the data for a certain period) and the significant effort involved in rerunning the test cases on a per-commit basis. In this paper, we present T-Evos, a dataset on test result and coverage evolution, covering 8,093 commits across 12 open-source Java projects. Our dataset includes the evolution of statement-level code coverage for every test case (either passed and failed), test result, all the builds information, code changes, and the corresponding bug reports. We conduct an initial analysis to demonstrate the overall dataset. In addition, we conduct an empirical study using T-Evos to study the characteristics of test failures in CI settings. We find that test failures are frequent, and while most failures are resolved within a day, some failures require several weeks to resolve. We highlight the relationship between code changes and test failure, and provide insights for future automated testing research. Our dataset may be used for future testing research and benchmarking in CI. Our findings provide an important first step in understanding code coverage evolution and test failures in a continuous environment.

Wed 13 Sep

Displayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change

15:30 - 17:00
Software Testing for Specialized Systems 1Research Papers / Journal-first Papers / NIER Track at Room C
Chair(s): Fabrizio Pastore University of Luxembourg
15:30
12m
Talk
DCLink: Bridging Data Constraint Changes and Implementations in FinTech Systems
Research Papers
Wensheng Tang Hong Kong University of Science and Technology, Chengpeng Wang Hong Kong University of Science and Technology, Peisen Yao Zhejing University, Rongxin Wu Xiamen University, Xianjin Fu Ant Group, Gang Fan Ant Group, Charles Zhang Hong Kong University of Science and Technology
File Attached
15:42
12m
Talk
Systematically Detecting Packet Validation Vulnerabilities in Embedded Network Stacks
Research Papers
Paschal Amusuo Purdue University, Ricardo Andrés Calvo Méndez Universidad Nacional de Colombia, Zhongwei Xu Xi'an JiaoTong University, Aravind Machiry Purdue University, James C. Davis Purdue University
Pre-print Media Attached File Attached
15:54
12m
Talk
WADIFF: A Differential Testing Framework for WebAssembly Runtimes
Research Papers
Shiyao Zhou The Hong Kong Polytechnic University, Muhui Jiang The Hong Kong Polytechnic University, Weimin Chen The Hong Kong Polytechnic University, Hao Zhou Hong Kong Polytechnic University, Haoyu Wang Huazhong University of Science and Technology, Xiapu Luo Hong Kong Polytechnic University
File Attached
16:06
12m
Talk
T-Evos: A Large-Scale Longitudinal Study on CI Test Execution and Failure
Journal-first Papers
An Ran Chen University of Alberta, Tse-Hsun (Peter) Chen Concordia University, Shaowei Wang University of Manitoba
Pre-print
16:18
12m
Talk
VRGuide: Efficient Testing of Virtual Reality Scenes via Dynamic Cut Coverage
Research Papers
Xiaoyin Wang University of Texas at San Antonio, Tahmid Rafi University of Texas at San Antonio, Na Meng Virginia Tech
File Attached
16:30
12m
Talk
PURLTL: Mining LTL Specification from Imperfect Traces in TestingRecorded talk
NIER Track
Bo Peng Sun Yat-Sen University, Pingjia Liang Sun Yat-Sen University, Tingchen Han Sun Yat-Sen University, Weilin Luo Sun Yat-Sen University, Jianfeng Du Guangdong University of Foreign Studies, Hai Wan School of Data and Computer Science, Sun Yat-sen University, Rongzhen Ye Sun Yat-Sen University, Yuhang Zheng Sun Yat-Sen University
Media Attached