Experimentation in organizations often starts with the marketing department hiring an agency to raise the ROI of their marketing efforts. Scaling experimentation in the whole organization leads to a larger quantity of experiments, preventing changes without testing, thus making better decisions, and reducing risk. However, we have learned that broadly scaling experimentation has negative effects on the marketing departments KPI of more revenue, which is preventing experimentation from scaling up.

The main objective of this paper is to make organizations aware of this Catch-22 when starting experimentation with the scope of raising ROI. We want to convince marketing departments, their agencies, and their vendors that if experimentation starts at marketing, then they need to create more awareness for the full power of experimentation by presenting all the results including the more common inconclusive and negative outcomes of experiments.

Our learnings suggest that this awareness must lead to budget and resources for a stand-alone experimentation Center of Excellence (CoE) outside of the marketing department, that different (not ROI focused) KPIs are needed when scaling up experimentation, and that specific organizational conditions must be met to maintain a future-proof set-up.