Database systems are widely used to store and query data. Test oracles have been proposed to find logic bugs in such systems, that is, bugs that cause the database system to compute an incorrect result. To realize a fully automated testing approach, such test oracles are paired with a test case generation technique; a test case refers to a database state and a query on which the test oracle can be applied. In this work, we propose the concept of Query Plan Guidance (QPG) for guiding automated testing towards “interesting” test cases. SQL and other query languages are declarative. Thus, to execute a query, the database system translates every operator in the source language to one of potentially many so-called physical operators that can be executed; the tree of physical operators is referred to as the query plan. Our intuition is that by steering testing towards exploring diverse query plans, we also explore more interesting behaviors—some of which are potentially incorrect. To this end, we propose a mutation technique that gradually applies promising mutations to the database state, causing the DBMS to create diverse query plans for subsequent queries. We applied our method to three mature, widely-used, and extensively-tested database systems—SQLite, TiDB, and CockroachDB—and found 53 unique, previously unknown bugs. Our method exercises 4.85—408.48× more unique query plans than a naive random generation method and 7.46× more than a code coverage guidance method. Since most database systems—including commercial ones—expose query plans to the user, we consider QPG a generally applicable, black-box approach and believe that the core idea could also be applied in other contexts (e.g., to measure the quality of a test suite).
Fri 19 MayDisplayed time zone: Hobart change
11:00 - 12:30 | Testing of database and low-level softwareTechnical Track / SEIP - Software Engineering in Practice / DEMO - Demonstrations / Journal-First Papers at Meeting Room 109 Chair(s): Michael Pradel University of Stuttgart | ||
11:00 15mTalk | Compiler Test-Program Generation via Memoized Configuration Search Technical Track Junjie Chen Tianjin University, Chenyao Suo College of Intelligence and Computing, Tianjin University, Jiajun Jiang Tianjin University, Peiqi Chen College of Intelligence and Computing, Tianjin University, Xingjian Li College of Intelligence and Computing, Tianjin University | ||
11:15 15mTalk | Generating Test Databases for Database-Backed Applications Technical Track | ||
11:30 15mTalk | Testing Database Engines via Query Plan Guidance Technical Track Pre-print | ||
11:45 15mTalk | Testing Database Systems via Differential Query Execution Technical Track Jiansen Song , Wensheng Dou Institute of Software Chinese Academy of Sciences, Ziyu Cui Institute of Software Chinese Academy of Sciences, Qianwang Dai Institute of Software Chinese Academy of Sciences, Wei Wang , Jun Wei Institute of Software at Chinese Academy of Sciences; University of Chinese Academy of Sciences; University of Chinese Academy of Sciences Chongqing School, Hua Zhong Institute of Software Chinese Academy of Sciences, Tao Huang Institute of Software Chinese Academy of Sciences Pre-print | ||
12:00 15mTalk | Achieving Last-Mile Functional Coverage in Testing Chip Design Software Implementations SEIP - Software Engineering in Practice Ming Yan College of Intelligence and Computing, Tianjin University, Junjie Chen Tianjin University, Hangyu Mao , Jiajun Jiang Tianjin University, Jianye Hao Tianjin University, Xingjian Li College of Intelligence and Computing, Tianjin University, Zhao Tian Tianjin University, Zhichao Chen , Dong Li , Dong Li , Yanwei Guo , Wulong Liu , Bin Wang , Yuefeng Sun , Yongshun Cui | ||
12:15 7mTalk | Randomized Differential Testing of RDF Stores DEMO - Demonstrations Rui Yang Institute of Software, Chinese Academy of Sciences, Yingying Zheng Institute of Software Chinese Academy of Sciences, Lei Tang Institute of Software Chinese Academy of Sciences, Wensheng Dou Institute of Software Chinese Academy of Sciences, Wei Wang , Jun Wei Institute of Software at Chinese Academy of Sciences; University of Chinese Academy of Sciences; University of Chinese Academy of Sciences Chongqing School Pre-print | ||
12:22 7mTalk | Boosting Compiler Testing via Compiler Optimization Exploration Journal-First Papers Junjie Chen Tianjin University, Chenyao Suo College of Intelligence and Computing, Tianjin University |