Continuous Testing is a term that has been adopted to describe different kinds of testing activities along a continuous delivery pipeline. Though those activities have been discussed in detail over the past decade, we found there is still much uncertainty as to how those ideas fit together, when they need to be executed, and what value they bring. Common questions include: “Is testing just a separate step in the pipeline?” and “How do we approach mapping the levels of the testing pyramid onto the pipeline stages?”

We will discuss testing features from ideation and coding to artifacts, deployment, and finally operating them in production. Though mapping testing activities onto a software delivery pipeline seems intuitive, we believe our model is more understandable. In our model, important concepts such as speed of feedback, building quality into the product, shift left/right, and the testing pyramid can be more easily understood.

Furthermore, we emphasize the often neglected, but profound impact of isolation levels on the software under test, in addition to the corresponding relationships between test types and the delivery pipeline. Mapping test types onto the pipeline is therefore guided by optimizing across three concerns: speed of feedback, test reliability, and overall confidence in the developed product.

We found that teams react well to this model and often report back that this approach has helped them demystify some of the buzzwords. Teams can also create a more cohesive delivery and testing strategy. Furthermore, this approach helps them establish a common vocabulary within a team and across teams. This aids in conversations with multiple stakeholders about testing activities during refinement and planning. In our experience, it has worked well with teams who are just beginning their Agile transformation, as well as those teams who are already well underway in their Agile journey (including those adopting a DevOps culture).