Understanding Bug-Reproducing Tests: A First Empirical Study
Developers create bug-reproducing tests that support debugging by failing as long as the bug is present, and passing once the bug has been fixed. These tests are usually integrated into existing test suites and executed regularly alongside all other tests to ensure that future regressions are caught. Despite this co-existence with other types of tests, the properties of bug-reproducing tests are scarcely researched, and it remains unclear whether they differ fundamentally. In this short paper, we provide an initial empirical study to understand bug-reproducing tests better. We analyze 642 bug-reproducing tests of 15 real-world Python systems. Overall, we find that bug-reproducing tests are not (statistically significantly) different from other tests regarding LOC, number of assertions, and complexity. However, bug-reproducing tests contain slightly more try/except blocks and “weak assertions” (e.g., assertNotEqual). Lastly, we detect that the majority (95%) of the bug-reproducing tests reproduce a single bug, while 5% reproduce multiple bugs. We conclude by discussing implications and future research directions.
Mon 13 AprDisplayed time zone: Brasilia, Distrito Federal, Brazil change
16:00 - 17:30 | Session 4: Test Automation and the Software ProcessAST 2026 at Oceania VI Chair(s): Phil McMinn University of Sheffield | ||
16:00 30mTalk | A Framework for Similarity-based and Resource-aware Orchestration of End-to-End Test Cases AST 2026 Cristian Augusto University of Oviedo, Antonia Bertolino Gran Sasso Science Institute, Guglielmo De Angelis CNR-IASI, Claudio de la Riva University of Oviedo, Francesca Lonetti CNR-ISTI, Jesús Morán University of Oviedo | ||
16:30 30mTalk | Understanding Bug-Reproducing Tests: A First Empirical Study AST 2026 Pre-print Media Attached | ||
17:00 30mTalk | Exploring Mocking Techniques for Managing External Dependencies in Service-Based Systems: A Mapping Study AST 2026 Benedito Fernando Albuquerque de Oliveira Federal University of Pernambuco, Fernando Castor University of Twente, Leo Fernandes Federal Institute of Alagoas (IFAL), Samuel Amorim IFAL/Brazil | ||