Write a Blog >>
ASE 2021
Sun 14 - Sat 20 November 2021 Australia
Wed 17 Nov 2021 11:00 - 11:20 at Koala - Large Scale Systems Chair(s): ingo Mueller

This paper is an experience paper.

For large-scale distributed systems, it is crucial to efficiently diagnose the root causes of incidents to maintain high system availability. The recent development of microservice architecture brings three major challenges (i.e., operation, system scale, and monitoring complexities) to root cause analysis (RCA) in industrial settings. To tackle these challenges, in this paper, we present Groot, an event-graph-based approach for RCA. Groot constructs a real-time causality graph based on events that summarize various types of metrics, logs, and activities in the system under analysis. Moreover, to incorporate domain knowledge from site reliability engineering (SRE) engineers, Groot can be customized with user-defined events and domain-specific rules. Currently, Groot is servicing for RCA among 5,000 services and is actively used by the SRE team in a global e-commerce system serving more than 185 million active buyers per year. Over 15 months, we collect a data set containing labeled root causes of 952 real production incidents for evaluation. The evaluation results show that Groot is able to achieve 95% top-3 accuracy and 78% top-1 accuracy. To share our experience in deploying and adopting RCA in industrial settings, we conduct survey to show that users of Groot find it helpful and easy to use. We also share the lessons learned from deploying and adopting Groot to solve RCA problems in production environments.

Wed 17 Nov

Displayed time zone: Hobart change

11:00 - 12:00
Large Scale SystemsIndustry Showcase / Research Papers at Koala
Chair(s): ingo Mueller Monash University
Groot: An Event-graph-based Approach for Root Cause Analysis in Industrial Settings
Research Papers
Hanzhang Wang eBay, Zhengkai Wu University of Illinois at Urbana-Champaign, Huai Jiang eBay, USA, Yichao Huang eBay, Jiamu Wang eBay, Selcuk Kopru eBay, Tao Xie Peking University
Lessons learned from hyper-parameter tuning for microservice candidate identification
Industry Showcase
Rahul Yedida North Carolina State University, Rahul Krishna IBM Research, Anup K. Kalia IBM Research, Tim Menzies North Carolina State University, Jin Xiao IBM Research, Maja Vukovic IBM Research
BeeSwarm: Enabling Parallel Scaling Performance Measurement in Continuous Integration for HPC Applications
Industry Showcase
Jacob Tronge Kent State University, qiang guan Kent State University, Jieyang Chen , Patricia Grubel Los Alamos National Laboratory, Tim Randles Los Alamos National Laboratory, Rusty Davis Los Alamos National Laboratory, Quincy Wofford Los Alamos National Laboratory, Steven Anaya Los Alamos National Laboratory