Replication-Free Failover: Protocol-Fenced Takeover for Stateful Services
We are the development and operations team of Alibaba Cloud’s messaging middleware, running thousands of tenant-isolated, stateful clusters in production. Through years of handling real incidents such as node crashes, I/O stalls, and network partitions, we observed a practical “trilemma” in stateful disaster recovery: low cost, high steady-state performance, and fast failover are hard to achieve at the same time, echoing well-known engineering trade-offs in distributed systems and SRE practice. This paper presents a production-grade, second-level failover mechanism for cloud-deployed stateful services, aiming to balance the three under realistic engineering constraints. Our design keeps the steady-state data path unchanged by continuing to use the local file system and cloud block storage. During failover, we enforce protocol-level fencing to enable fast and safe takeover: leveraging disk Multi-Attach and NVMe Persistent Reservation, the new node can preempt immediately to prevent split-brain without waiting for the detach/attach workflow. We further introduce decentralized lease probing and a least-privilege, node-side agent that performs preemption, mounting, and switchover orchestration on the critical recovery path. Experiments using Apache RocketMQ as a representative workload show that, without reserving extra physical replicas and without introducing additional write amplification, our approach achieves end-to-end failover in seconds while keeping steady-state throughput and latency close to a native single-replica deployment. The system has completed tens of thousands of disaster-recovery drills and is now running in production.
Thu 9 JulDisplayed time zone: Eastern Time (US & Canada) change
10:30 - 12:30 | Service and SystemsIndustry Papers / Research Papers / Journal-First Paper at MB 2.435 Chair(s): Seulbae Kim Pohang University of Science and Technology | ||
10:30 20mTalk | LLM Agents for AIOps in Kubernetes: An Industrial Experience Report with Red Hat OpenShift Industry Papers | ||
10:50 20mTalk | Replication-Free Failover: Protocol-Fenced Takeover for Stateful Services Industry Papers Rongtong Jin Alibaba Cloud Computing Co. Ltd., Yinyou Gu Alibaba Cloud Computing Co. Ltd., Juntao Ji Alibaba Cloud Computing Co. Ltd., Yubao Fu Alibaba Cloud Computing Co. Ltd., Tao Liu Alibaba Cloud Computing, Fuzhi Lai Alibaba Cloud Computing, Gaoyang Cai Alibaba Cloud Computing, Qingshan Lin Alibaba Cloud Computing Co. Ltd. | ||
11:10 20mTalk | Programming of Automation Configuration in Smart Home Systems: Challenges and Opportunities Journal-First Paper Sheik Murad Hassan Anik Virginia Tech, Xinghua Gao Virginia Tech, Hao Zhong Shanghai Jiao Tong University, Xiaoyin Wang University of Texas at San Antonio, Na Meng Virginia Tech Link to publication DOI Pre-print | ||
11:30 20mTalk | SmartDispatch: Dynamic Substitution of NumPy-style APIs on Heterogenous CPU-GPU Systems Research Papers Jinku Cui North Carolina State University, Yueming Hao Meta, Shuyin Jiao North Carolina State University, Jiajia Li North Carolina State University, Xu Liu North Carolina State University | ||
11:50 20mTalk | CodeCureAgent: Automatic Classification and Repair of Static Analysis Warnings Research Papers Pascal Joos CISPA Helmholtz Center for Information Security, Islem BOUZENIA CISPA Helmholtz Center for Information Security, Michael Pradel CISPA Helmholtz Center for Information Security DOI | ||
12:10 20mTalk | SwarmBox: A Plug-and-Play Drone Swarm Framework for Streamlined Development and Comprehensive Analysis Research Papers Minki Lee Pohang University of Science and Technology, Seojin Lee Daegu Gyeongbuk Institute of Science and Technology, Seulbae Kim Pohang University of Science and Technology DOI Pre-print Media Attached | ||