Running a Red Light: An Investigation into Why Software Engineers (Occasionally) Ignore Coverage Checks
Many modern code coverage tools track and report code coverage data generated from running tests during continuous integration. They report code coverage data through a variety of channels, including email, Slack, Mattermost, or through the web interface of social coding platforms such as GitHub. In fact, this ensemble of tools can be configured in such a way that the software engineer gets a failing status check when code coverage drops below a certain threshold. In this study, we broadly investigate the opinions and experience with code coverage tools through a survey among 279 software engineers whose projects use the Codecov coverage tool and bot. In particular, we are investigating why software engineers would ignore a failing status check caused by drop in code coverage. We observe that >80% of software engineers — at least sometimes — ignore these failing status checks, and we get insights into the main reasons why software engineers ignore these checks.
Mon 15 AprDisplayed time zone: Lisbon change
11:00 - 12:30 | Session 1: Test CoverageAST 2024 at Amália Rodrigues Chair(s): Gilles Perrouin Fonds de la Recherche Scientifique - FNRS & University of Namur | ||
11:00 20mFull-paper | Mutation Coverage is not Strongly Correlated with Mutation Coverage AST 2024 | ||
11:20 20mFull-paper | Running a Red Light: An Investigation into Why Software Engineers (Occasionally) Ignore Coverage Checks AST 2024 Alexander Sterk Delft University of Technology, Mairieli Wessel Radboud University, Eli Hooten Sentry.io, Andy Zaidman Delft University of Technology DOI Pre-print | ||
11:40 20mFull-paper | Coverage-based Strategies for the Automated Synthesis of Test Scenarios for Conversational Agents AST 2024 Pablo C Canizares Autonomous University of Madrid, Spain, Daniel Ávila Autonomous University of Madrid, Sara Perez-Soler Universidad Autónoma de Madrid, Esther Guerra Universidad Autónoma de Madrid, Juan de Lara Autonomous University of Madrid Pre-print | ||
12:00 20mFull-paper | WallMauer: Robust Code Coverage Instrumentation for Android Apps AST 2024 Michael Auer University of Passau, Iván Arcuschin Moreno University of Buenos Aires, Argentina, Gordon Fraser University of Passau |