The Magic of Statistics for Software Testing: How to Foresee the Unseen

Ensuring software correctness is essential as software increasingly governs critical aspects of modern life. Formal methods for program verification, while powerful, often struggle with scalability when faced with the complexity of modern systems. Meanwhile, software testing—finding defects by executing the program—is practical but inherently incomplete, as it inevitably misses certain behaviors, i.e., the “unseens,” leaving critical gaps in verification.

In this tutorial, I illuminate the transformative potential of statistical methods in addressing these challenges, with a particular focus on residual risk analysis. Residual risk analysis quantifies the likelihood of undiscovered bugs remaining in the software after testing by estimating the probability of finding a new, previously unseen bug in the next test input.

We will begin by demonstrating how statistical estimators can assess residual risk using records from software testing—such as code coverage data—through a hands-on example. The tutorial then explores several advanced extensions to adapt residual risk analysis for more realistic testing scenarios. By the end of this session, participants will gain a deeper understanding of how statistical thinking can provide actionable insights into the unseen behaviors of software systems, ultimately making testing more accountable, transparent, and efficient.