EASE 2022
Mon 13 - Wed 15 June 2022 Göteborg, Sweden

Background: Sample Ratio Mismatch (SRM) checks can help detect data quality issues in online experimentation. Not all experimentation platforms provide these checks as part of their solution. Users of these platforms must therefore manually check for SRM, or rely on additional processes—such as checklists—or automation.

Objective: To ensure reliable and early detection of SRM, we wanted to automate the detection and analysis of SRM in experiments running on third-party experimentation platforms.

Method: A set of Looker dashboards were built to facilitate self-serve SRM detection and root cause analysis. In addition, we added email and chat based alerting to pro-actively inform experimenters of SRM and guide them towards these dashboards when needed.

Results: Several cases of SRM have been detected and experimenters have been warned. Bad decisions based on flawed data were avoided. We provide one such example as an illustration.

Conclusions: SRM checks are relatively straightforward to automate and can be useful for data quality monitoring even for companies who rely on third-party experimentation platforms. Pro-active alerting—rather than passive reporting—can reduce time to detection and help non-experts avoid making decisions based on biased data.

Wed 15 Jun

Displayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change

15:15 - 15:30
Industrial track session 2Industrial Track at Wednesday Track 1 session 2 (Kelvin and online)
Chair(s): Jocelyn Simmonds University of Chile
15:15
15m
Talk
Automated Sample Ratio Mismatch (SRM) detection and analysis
Industrial Track
Lukas Vermeer Vista, Kevin Anderson Delft University of Technology, Mauricio Acebal Vista
Link to publication DOI

Information for Participants
Wed 15 Jun 2022 15:15 - 15:30 at Wednesday Track 1 session 2 (Kelvin and online) - Industrial track session 2 Chair(s): Jocelyn Simmonds
Info for room Wednesday Track 1 session 2 (Kelvin and online):

Link to join: https://eu01web.zoom.us/j/67762559229?pwd=Z1RmcDBRUWdjY04vcUxFUVovM1BBZz09