Seeding and Mocking in White-Box Fuzzing Enterprise RPC APIs: An Industrial Case Study
Microservices is now becoming a promising architecture to build large-scale web services in industry. Due to the high complexity of enterprise microservices, industry has an urgent need to have a solution to enable automated testing of such systems. EvoMaster is an open-source fuzzer, equipped with the state-of-the-art techniques for supporting automated system-level testing of Web APIs. It has been assessed as the most performant tool in two recent empirical studies in terms of line coverage and fault detection. In this paper, we carried out an empirical experiment to investigate how to better apply the state-of-the-art academic prototype (i.e., EvoMaster) in industrial context. We extended the tool to handle seeding of existing industrial tests, and mocking of external services with their data handled as part of the input fuzzing. We studied two configurations of EvoMaster, using two time budgets, on 40 enterprise RPC-based APIs (involving 5.6 million lines of code for their core business logic) at Meituan Results show that, compared to existing practice of manual system-level testing and tests produced by record and replay of online traffic, EvoMaster demonstrates clear additional benefits. EvoMaster with the best configuration is capable of covering up to 32.4% line coverage, covering more than 10% line coverage on 36 out of 40 (90%) case studies, and identifying on average 3520 potential faults in these 40 APIs. In addition, we also identified and discussed important challenges in fuzzing enterprise microservices that must be addressed in the future.
Wed 30 OctDisplayed time zone: Pacific Time (US & Canada) change
15:30 - 16:30 | Fuzzing 2Industry Showcase / NIER Track / Research Papers at Magnoila Chair(s): Marcel Böhme MPI-SP, Bochum | ||
15:30 15mTalk | Differential Fuzzing for Data Distribution Service Programs with Dynamic Configuration Research Papers Dohyun Ryu The Pennsylvania State University, Giyeol Kim The Pennsylvania State University, Daeun Lee Pusan National University, Seongjin Kim The Pennsylvania State University, Seungjin Bae The Pennsylvania State University, Junghwan Rhee University of Central Oklahoma, Taegyu Kim The Pennsylvania State University | ||
15:45 15mTalk | Seeding and Mocking in White-Box Fuzzing Enterprise RPC APIs: An Industrial Case Study Industry Showcase Man Zhang Beihang University, China, Andrea Arcuri Kristiania University College and Oslo Metropolitan University, Piyun Teng Meituan, kaiming.xue Meituan, Wenhao Wang Meituan | ||
16:00 15mTalk | Industry Practice of Directed Kernel Fuzzing for Open-source Linux Distribution Industry Showcase Heyuan Shi Central South University, Shijun chen Central South University, Runzhe Wang Alibaba Group, Yuhan Chen Central South Sniversity, Weibo Zhang Central South University, Qiang Zhang Hunan University, Yuheng Shen Tsinghua University, Xiaohai Shi Alibaba Group, Chao Hu Central South University, Yu Jiang Tsinghua University | ||
16:15 10mTalk | Visualizing and Understanding the Internals of Fuzzing NIER Track Sriteja Kummita Fraunhofer Institute for Mechatronic Systems Design (Fraunhofer IEM), Zenong Zhang The University of Texas - Dallas, Eric Bodden Heinz Nixdorf Institut, Paderborn University and Fraunhofer IEM, Shiyi Wei University of Texas at Dallas |