Are We Over-Testing? Studying Redundant Test Executions in Multi-Job CI Workflows
Continuous Integration (CI) has become an integral part of modern software development, with platforms like GitHub Actions (GHA) enabling automated testing across multiple CI jobs targeting different environments (e.g., operating systems or language versions). However, many test cases often yield identical outcomes (pass or fail) across jobs, which can lead to redundant executions, longer workflows, and inefficient resource usage. For example, a typical GHA workflow might rerun the same test set on Linux, Windows, and macOS jobs, despite producing similar results. In this study, we aim to investigate how test cases are distributed across jobs in multi-job GHA workflows and examine the extent and consistency of redundant test executions. To achieve our objective, we will analyze a large dataset of open-source GitHub projects using GHA, employing static analysis of YAML configuration files and logs to identify test allocation, overlapping patterns, test outcome consistency, and estimate potential time savings. The findings of this study will offer insight into current testing practices in GHA and suggest opportunities to optimize workflows by reducing unnecessary duplication, improving both resource and cost efficiency.
Fri 12 SepDisplayed time zone: Auckland, Wellington change
15:30 - 16:30 | Session 18 - Quality Assurance 3Industry Track / Research Papers Track at Case Room 2 260-057 Chair(s): Raula Gaikovina Kula The University of Osaka | ||
15:30 15mFull-paper | Evaluation of the Language Server Protocol for Static Dependency Analysis Research Papers Track Falko Galperin Axivion GmbH, Michel Krause Universtität Bremen, Rainer Koschke University of Bremen Pre-print | ||
15:55 15m | Monitoring Continuous Integration Practices in Industry: A Case Study Industry Track Jadson Santos Universidade Federal do Rio Grande do Norte, Daniel Alencar Da Costa University of Otago, Uirá Kulesza Federal University of Rio Grande do Norte | ||
16:10 15m | Constraint Discovery for Structured Generation via LLM-Guided SMT Inference Industry Track Hrishikesh Karmarkar TCS Research, Siddhesh Pagar TCS Research, Supriya Agrawal Tata Consultancy Services Ltd. (TCS), Vaibhavi Joshi TCS Research, Naman Paul TCS Research, Sagar Verma Tata Consultancy Services Ltd. (TCS) | ||